February 3, 2002 Version 1.8604 Sigh. Integrating CAB's cookie management may have been a bad idea, it doesn't handle domain cookies at all. Had to revert to always using the OVL's management. Also fixed a bug with redirects causing a previously cached URL to be overwritten. Also a minor tweak, the "Resolving Host" status message really means something now. The OVL used to jump straight to the "Connecting to Host" status message, even though it was just starting the gethostbyname() call. Now if you see "Resolving Host" for a long time, that probably means the destination's domain is down. February 3, 2002 Version 1.8603 Fixed a memory leak in the gzip support code. Fixed an unterminated string in the authentication code. Fixed the make_unix_time function, which was using local time instead of GMT. Added support for CAB's cookie functions. When the version of CAB supports cookies, the OVL gives them to CAB instead of storing them itself. Exception - the OVL always keeps single-session cookies to itself (cookies that have no expiration date, and thus expire when the browser terminates). When CAB does not support cookies, the OVL does all the cookie management, as before. Tweaked the Makefile to make my life easier building SSL vs non-SSL binaries... December 5, 2001 Version 1.8602 Fixed three unrelated bugs. One caused redirects to sometimes be processed incorrectly. Another one caused some web servers to hang instead of responding; this was because the overlay was sending lines terminated by '\n' instead of the expected "\r\n" line terminator. Finally, the gzip handling was changed to only decompress files of type text/*. This should fix the problems with downloading .tar.gz files and such. November 23, 2001 Version 1.8601 The overlay only works if drive U is the current drive. Added code to set the current drive and path if it's not starting on U:. November 21, 2001 Version 1.86 Now uses libz instead of spawning external gzip app. Greatly improves HTML download speed when the server supports it. November 11, 2001 Version 1.8406 Updated to work with MiNTlib 0.57. My entire system has been upgraded with the SpareMiNT distribution. Also a minor improvement, for HTTP/1.1 servers assume the connection is kept open by default. Connection re-use is much more effective now but there is still room for improvement. Note - because of the particular way that CAB loads modules, this code is still very much dependent on how the C library initializes itself. The code in init.c is extremely library-dependent and there's really no way to avoid this. The linking process also cannot use the standard C runtime startup, so things are still inconvenient there. Since the SpareMiNT distribution only includes a 68020 version of the OpenSSL libraries, this binary is 68020-only. When I get a chance to rebuild the OpenSSL library I will build a 68000 binary. February 13, 1998 Version 1.8404 (1.8403 was only sent to one beta tester.) Fixed a bug in redirect processing, would resolve the wrong path if the "http:" prefix was missing. Fixed a bug in the main request handler, it didn't allow POSTing form results via mailto. Fixed a refresh bug, the reloadflag that CAB gave us was declared as int but needed to be short. Fixed an SSL bug, have to run the library in bug-compatibility mode to talk to certain broken SSL servers. Doc note - if you have an accelerated machine (Nemesis, Afterburner, etc.) you will probably experience lots of crashes unless you turn off the Fast RAM flags on the CAB.OVL binary. Don't know why, but turning off the flags makes things more stable. December 27, 1997 Fixed bug with CAB 1.5, wasn't giving back HEADer information. Removed code to map HTTP errors to system error codes; we just report "success" and let CAB display the error message. Added X-Mailer header to the mail handler. This is version 1.8402. The last two digits of the version number are now used as a fix/patch counter. The December 20 release was 1.8401. December 20, 1997 Added a first attempt at general documentation (ovldoc.txt). Fixed a bug in the debug(!) printing. Reduced frequency of byte count updates, they were actually slowing down transfers with a fast connection. December 11, 1997 Found several bugs in the authorization code. It now passes all of the basic auth tests on http://digest-test.agranat.com. (I'm pretty happy with this, even the latest version of Netscape fails the multiple-realm test.) Also found bugs in the config file handling; it now looks for "modules\cabovl.cnf" in addition to "cabovl.cnf", since you will probably want the OVL in your modules subdirectory. It occurred to me that since you could have multiple copies of the OVL loaded, depending on use of other CAB modules, this OVL ought to be a shared-text binary. We'll try that for 1.85. Found a problem with NCSA web servers, they freak out if you send them an HTTP/1.1 request; they send back a response without a valid HTTP version ID. The OVL now recognizes this case. Also found a problem with Netcache 3.11d proxy server on Solaris, it can insert extra bytes into Chunked text transfers. (God knows if it also happens on binary files; I can't verify the content!) The OVL now tries to detect these extra inserted bytes and save them into the output file. Re-enabled ftp, gopher, wais, and nntp via HTTP-based proxy. Silly to have turned that off in the first place. Added nil-pointer checking back in for CAB proxy table. The 2.7 beta is giving nil-pointers where it didn't used to, and shouldn't... December 7, 1997 Finished Basic authorization. I was going to work on Digest auth, but the HTTP/1.1 spec hasn't really been finalized yet. A new 1.1 draft was issued on November 21 and there are still a couple of open issues unresolved in the spec. Also I'm sure it will be some months before anyone has web servers that actually use this stuff. Fixed a bug with redirects, they needed to be fully parsed before handing them back to CAB. Also, redirects used to involve nested recursive calls to the GetUrl function, potentially consuming a lot of stack space. They now use a longjmp into GetUrl, so no additional stack space is consumed. Added maxhosts and maxnodes keywords to the config file. This lets you tune the size of the in-memory URL info cache. Lower bound is "2" for both of these values. See the CABOVL.CNF file. I'm still waiting on supporting the CAB 2.7 cookie management, it looks like the current draft is missing some functionality. So, the OVL continues to do all cookie management by itself. At this point I need to stop and take care of other things in my life. The features I see that still need to be added are: internal Gopher - later internal WAIS - much later. I haven't used WAIS in ages... piped downloads - it can be done in the OVL, clumsily, or it could be done in the filesystem, elegantly. Also, it could be done just in the minixfs driver, or it could be done in the MiNT kernel, thus supporting all filesystems. I'm beginning to lean toward changing the main kernel. December 6, 1997 Finished cookie support. Preliminary authentication/authorization is in, but it's not really decent yet. This is version 1.84. Note - Alex has added some kind of cookie caching mechanism to CAB 2.7. I have not added support for that yet. Currently the overlay caches cookies in memory only, and it doesn't check for the new functions in CAB 2.7. That'll come later. Also, I got rid of the CMGZIP and MAILUSER environment variables. I now have a CABOVL.CNF config file. Currently there are only 3 keywords recognized for this file: mailfrom, gzip, and cookies. See the sample file for descriptions. The CABOVL.CNF file goes in the same folder as CAB.APP and CAB.OVL. November 30, 1997 Did some major rewriting and reorganizing here. I guess that'll be obvious if you examine the source files. There still isn't much in the way of user- visible changes. But underneath, the thing is fully HTTP/1.1 compliant, and now supports reusable connections. It also fixes some parts where I hadn't followed the HTTP/1.1 spec carefully enough before. Cookies and Authentication/Authorization are still missing; other work took too long to complete. Maybe by Christmas... PS: I'm calling this version 1.83 now; the July code with SSL was 1.82 and was never publically distributed. July 6, 1997 The overlay can now support HTTPS for secure connections using SSL. The support is all conditionally compiled in the network.c file; you will have to build the SSLeay library yourself to use it. I used SSLeay 0.80, 0.81 is already out and there are mmany patches to that as well. I haven't built with 0.81 yet. I cannot distribute the SSL binaries due to stupid US laws about exporting encryption technology. Some kind soul needs to contact me about providing SSL-capable binaries from outside the US. June 17, 1997 I received a new CAB 2.5 beta from Alexander Clauss which fixed the parameters of get_url_if_modified to agree with the documentation. So, I had to change my definition again. No other functional changes. June 14, 1997 OK, I swear this is the real, final, 1.81. Fixed the bytes-received reporting. Before, you could sometimes see status like "received 300/250" or other nonsense, because of the way the overlay had sent the CAB parent a status message, and then continued processing. Since CAB gives the overlay pointers to work with and update, the overlay could have updated the pointers multiple times before CAB got around to processing all of the status messages. Now it will not update the filesize pointer until after all status messages have been processed. (Gee, never thought a program could work *too* fast now, did you...) Broke out the time/date stamp version stuff into a separate file, for easier updating. Stripped down the child startup code. Rewrote gzip spawning code. On aborts, return 0 (success) instead of -1. CAB seems to like this better. June 13, 1997 I believe that I've finally wiped out the STOP button crashes. There was also an error in the CAB 2.5 documentation, the last argument to get_url_if_modified is the time itself, not a pointer to the time. The get_url_if_modified function now works with the latest CAB 2.5 beta. I've changed my mind about the Cookie management, and have decided to release this 1.81 as-is. Further additions will have to wait. One more tweak - if a HEAD/get_url_info is redirected, the info is saved so that the subsequent GET/get_url on the same url doesn't do a redundant query. June 11, 1997 This is version 1.81. 1.80 was accidentally released with 68020 libraries linked in, thus causing it to crash on unmodified STs. There was also a bug where hitting the STOP button during a transfer would very likely crash the overlay. Also, the overlay was calling menu_register unconditionally, when it should have only done it if MultiTOS was active. The overlay now includes Accept and Host headers in its requests, to allow it to talk to HTTP 1.1 servers. I'm still leaving "HTTP/1.0" for my request version, since I haven't fully digested the HTTP 1.1 spec. I have also added support for the new get_url_if_modified call in CAB 2.0. At this time, CAB 2.0 is broken and does not use the new call, but that should be fixed soon. The overlay also uses a new CMGZIP environment variable, which should be set to the path of your gzip executable. If set, the overlay will add an Accept-Encoding: x-gzip, x-compress, gzip, compress header to all of its requests. Any data that is received with a header of Content-Encoding: gzip or whatever will be piped through gzip and uncompressed before storing to disk. There is a separate bug in the CAB 2.0 beta, where pressing the STOP button during a transfer corrupts CAB's memory. This will cause the overlay to crash. The bug has not appeared in CAB 1.5 (so far). This is not the final 1.81 release. I plan to add Cookie management before making a general 1.81 release available. Please bash on this some more and let me know what breaks. June 4, 1997 This code is derived from Andrew Vesperman's version 1.64 source release. However, I have radically restructured a lot of the code to fit it into a multi-threaded model. I have fixed old bugs in the 1.64 code, and no doubt introduced several new ones of my own. This module is numbered version 1.80, just to keep it separate from the releases that Andrew did. His last release was 1.77, which included partial FTP support. That code is not present in this release. This release includes mailto support as well as FTP support via an FTP proxy. Real FTP isn't done yet. Since I didn't have access to the 1.7x source code, I have to write it again from scratch. DETAILS: This overlay module works with CAB 1.5 and the CAB 2.0 demo. It is known to have problems with the CAB 2.0 beta that is currently out. This module is different from others because it is implemented as a threaded subprocess, and does most of its work concurrently with the main CAB process. The overlay has been tested with MiNTnet 1.01 and multiple MiNT versions from 1.12 up. It also works with MultiTOS (AES 4.1) other AES replacements, but is known to crash under Geneva. (If this is a problem for you, feel free to dig through the source code and fix it. I don't have Geneva, so I won't be looking for the problem myself.) The overlay uses two environment variables to alter its behavior. If the CMDEBUG variable is set, the overlay will log its activity into a file called cmdebug.log. The other variable is the MAILUSER variable, which should be set to your email address. This setting is used by the mailto processing to set your return address on outgoing messages. FEATURES: Faster URL-info caching Faster redirect handling Faster transfers Supports HTTP 1.1 error codes ...did I mention that it's faster? Better proxy handling - uses the CAB proxy variables instead of environment variables Thanks to my beta testers who volunteered to try this code in its early stages and suffered through its evolution over the past couple weeks: Yves Pelletier Anders Eriksson Katherine L Ellis Jo Even Skarstein Rodolphe Pineau atack Mario Becroft Bill Hallman This code could not have been released without their input. -- Howard Chu hyc@locus.com, chu@platinum.com PS: since this overlay spawns a child process that accesses the parent's memory, you must set the program flags on CAB.APP to be Global, or run MiNT without any memory protection at all. Btw, I'm using a fairly ancient release of MiNT (1.12, with patches) and MiNTnet 1.01, and pppd 2.1.2, patched to support 115200bps connections, on HSModem0.7. PPS: the included source code was built with the patchlevel 44 MiNT library. The code in init.c is highly dependent on the library version, and will only compile under GCC. Yes, it's ugly, and sane programmers don't go mucking about in their C library like this, but I felt it was necessary to obtain the fastest performance with the smallest memory consumption. Future: I figure it needs a real FTP client. Will also add support for the get_url_if_modified function in CAB 2.0; this will speed up response time even more. Still working out how to parallelize things more, and boost the network efficiency some more. Unfortunately I have no HTTP 1.1 servers available to test with. gzip/compress decoding might be worth adding too.