Project Hiper Schedule and Dates

Related:
Roadmap

I'll try to add info here as I proceed. If you want to comment on anything here, or provide some other kind of feedback on the Hiper work, please post all such to the curl-library mailing list!

Dec 6 2005 I committed my currently used code to CVS: cool.haxx.se/cvs.cgi/curl/hiper/ - not terribly much to see but I wanted to added it to make it easier for me to keep older versions around. This doesn't include the (minor) changes to libcurl I've done so far but none of those changes are required for this to run/work.

Dec 6 2005 Ok, I think I'll change my measuring approach and I will instead write up my own HTTP server and control that to allow it to either sit perfectly quiet (just keeping the connection alive) or to send an endless stream of data. By allowing the selection of behaviour be based on the URL, I should be able to modify my test program to run and test a given number of "active" connections with a given number of "existing connections". I guess the real-world URLs are too hard to use to make adequate tests with.
I found this little benchmark on select() performance compared to other event systems:

Dec 6 2005 Been struggling to get a test program that can be used to measure and show the current overhead/speed in a good way. I've been quite surprised by the speed of the existing implementation. My test application easily initiates and transfers 2000 simultaneous transfers (and man does this work nicer with a c-ares built libcurl!). The average time spent for each curl_multi_perform() in these cases is easily measured, but it is not easily to draw any specific conclusions. With 50 simultaneous connections, the average time per invoke is less than 600us on my Athlon XP2800 Linux 2.6/glibc 2.3.5/libcurl 7.15.1 test setup.
Some more numbers from the tests. I have a list with a few thousand URLs that the test app reads from and starts transferring data from. See below for more descriptions.

Total time 108149477us - Paused 12906983us = Active 95242494us = Active/total 47621us Active/(connections that delivered data) = 115585us 824 out of 2000 connections provided data 46684 calls to curl_multi_perform(), average 206 alive. Average time: 2040us 45267 calls to select(), average 193 alive Average number of readable connections per select() return: 172 Max number of readable connections for a single select() return: 549 0 select() timeouts

Total time - duration of the test. It stops when less than 50 connections are alive
Paused - time we don't care about, during which select() is called
Active - time we care about. The total amount spent in curl_multi_perform()
Active/total - is then the active time split on total number of connections (2000 in this case)
It is worth noting that not during the test, not a single select() call timed out. I use a 50ms timeout
The numbers shown here are quoted from a single test run, but they are very similar on repeated invokes

I managed to get this very high number of simultaneous connections returned from the select() call by patching libcurl to only read from connections one byte at a time!
The test program finally produces a summary that is separated into number of alive connections. So that we can see the average time curl_multi_perform() takes when N connections are still alive. I'll show a few lines from the list (times are in microseconds):
Time 50 connections, average 579 max 2351 (283 laps) average/conn: 11 Time 60 connections, average 673 max 3036 (594 laps) average/conn: 11 Time 70 connections, average 877 max 5402 (566 laps) average/conn: 12 Time 100 connections, average 1134 max 2367 (431 laps) average/conn: 11 Time 150 connections, average 1787 max 3025 (11 laps) average/conn: 11 Time 200 connections, average 2112 max 3843 (74 laps) average/conn: 10 Time 300 connections, average 2901 max 5629 (27 laps) average/conn: 9 Time 401 connections, average 3855 max 8623 (65 laps) average/conn: 9 Time 500 connections, average 4552 max 8409 (25 laps) average/conn: 9 Time 915 connections, average 11700 max 38587 (6 laps) average/conn: 12

Worth noticing is that this splits the time on "alive connections", while we don't really know how many connections that are readable at any given time here. The average number shown above does indicate that the amount of readable connections are fairly high all the time though.

Dec 2 2005 Made a script that figuered out a few thousand "random" URLs.
Started writing a test application using the existing multi interface. It fetches N simultaneous URLs and measures how long time is spent in the curl_multi_perform() calls. With "just" 100 simultaneous downloads the time spent is very small so I guess I need to up the timer resolution and use many more simultaneous transfers. Since my goal is to increase performance when very many transfers are in progress I need to somehow weight the stats counters to consider the amount of current transfers.
I've decided I'll commit/shouw my test program(s) as soon as I have something that actually show numbers I trust and are reliable.

Dec 1 2005 Daniel started working full-time on project Hiper (during the entire month of December).

Nov 29 2005 Mailed the first quarterly report in Swedish to IIS, about project Hiper. (282KB PDF)

Nov 28 2005 Discussing how the "zero copy" interface for libcurl could work. On the curl-library mailing list.

Nov 9 2005 Discussing possible Windows-adjustments to the hiper concept on the curl-library mailing list.

Nov 4 2005 Added the API and Sample app pages here.

Nov 3 2005 announced the 'hiper' project to the world, and added this hiper section of the curl web site.

Oct 24 2005 Daniel visited the "Internetdagarna" conference in Stockholm, Sweden and receives a diploma, a flower and some sturdy handshakes when the IIS foundation announced that Daniel will receive 150,000 SEK from their fund. Daniel was notified about their decision roughly one week before this.

Page updated December 06, 2005.
web site info