$Id: ChangeLog,v 1.29 2003/07/02 02:49:39 mmondor Exp $ Release: mmftpd 0.0.17 devl Date : July 1, 2003 By : Matthew Mondor * Bug fixes - The IP-address based connection rate and DNS cache expire thread used to sometimes waste CPU cycles looping more often that it should, sleeping for 0 seconds between loops. Fixed. - mmstatd(8) had a bugfix and was upgraded to 0.0.8 - If the specified port range for passive data connections was small and that alot of transfers would currently being served, it was possible to not be able to bind(2) a new socket. This is less likely now that mmftpd uses 4.2BSD setsockopt(2) to set the SO_REUSEADDR flag on the socket. The new system will also wait up to a maximum of 30 seconds while performing up to 64 retries per second on unsuccessful bind(2) attempts, running among ports within the allowed range. The previous system would simply always fail after 64 tries. - Because mmftpd used to enclose alot of mmstat() operations into a transaction using mmstat_transact() it was possible to have problems sending the request to the mmstatd(8) server on some systems where the default AF_UNIX DGRAM socket buffer was too small. In such circumstances the "connection" to the server would be reset and re-established, which in turn could cause problems if mmftpd was set to chroot(2) at startup and the mmstatd(8) socket was no longer available from within the jail. 4.2BSD setsockopt(2) is now used by mmstat(3) and mmstatd(8) to solve this issue. * Important - The mmstat(3) key names were modified to be '|' separated rather than '.' separated. Although I have been trying to avoid such a change which obviously requires fixing alot of scripts I am using, it was proven with time that the '.' character was too widely used and that '|' was ideal as a replacement. It is not hard to use a script to use the mmstat(8) reset command and convert all old entries to the new type if necessary. As a result, filenames which comport '.' characters for which counters are maintained are much better to handle and to isolate. The same applies to IP addresses. - Per-user mmstat statistics are now formated as mmftpd|user||* instead of mmftpd..* so that it is easier to isolate the actual user statistics from the total ones from scripts. - When per-file download statistics are wanted, the new mmstat key format becomes mmftpd|user||GET| instead of the previous mmftpd.. - The maximum key name length now was bumped from 128 to 256 bytes. As a result, a special log file synchronization will be required as was the case for mmstatd 0.0.7 upgrade (and mmftpd 0.0.16 upgrade). This also requires mmstat clients using the mmstat(3) API to be recompiled. Release: mmftpd 0.0.16 devl Date : June 19, 2003 By : Matthew Mondor * Important - Important changes to mmstat(3) and mmstatd(8) were made which require a database synchronization procedure to be performed before upgrading. See mmstatd/ChangeLog for details. - The GROUPS directive in mmftpd.conf(5) and mmstatd.conf(5) now expect groups to be comma-separated instead of space-separated. The quotes then of course also become optional if multiple groups are specified. This was made because mmreadcfg(3) library is also used by other of my projects for which comma-separated groups were also required. * Miscelaneous new features - When user has no rights to modify or upload (i.e. usually anonymous user) all files now appear as read-only and non-executable, and directories as read-only, when using LIST. - mmftpd now reports the number of bytes transfered in both directions for each user via the mmstat(3) facility. - Via the mmstat(3) facility can also be optionally recorded various failures such as bad logins, permission denied errors, illegal PORT, etc. - The new configuration file keywords PASV_RANGE, PASV_RANGE_MIN and PASV_RANGE_MAX were added which permit to limit mmftpd to use a specified port range for the PASV/LPSV/EPSV commands. This can also be used to randomize the port number used within the normal allowed high port range rather than letting the system automatically choose a free port. * Bug fixes - Fixed ALIGN_CEIL alignment macros which used to always add bytes even if the value was already aligned properly. A few bytes were often wasted as the result of this. - The STAT command would not properly format the off_t type on systems such as Linux/i386 on which off_t is 32-bit. This was fixed by using type casting to the expected 64-bit type. - On systems using glibc (such as Linux), the headerfile has to be explicitely included to use the crypt(3) function. - There is an issue where some versions of glibc do not define nfds_t in . mmlib/mmfd.h was introduced a basic check for this, which may be uncommented if this occurs. * Other - The FIFO library performance was improved and mmftpd now allocates FIFO buffers more efficiently using a dynamic pool, it used to be one of rare occations where malloc()/free() still had to be called at connection startup and disconnection. These FIFOs are used to remember which directories were visited recently to not flood the user with README file contents, when enabled. - Interface of various lower layer functions and macros were improved, and mmftpd was migrated to use the new ones. - Where appropriate and possible, const keyword was added to the function parameters and global data types. - Some more source cleanups - mmstatd will now perform much better when used with a large number of keys, and will now retain the real time of the operation if it had to be killed before it can incorporate the recovery logs and sync. - Now uses mmhash(3) for user lookup which should make mmftpd faster when many simultaneous users are logged in at the same time (this is used to verify user login limits and link a new user to the shared tree quota structure). - mmhash(3) is now used by mmserver(3) as well, which should enhance connection validity and DNS hostname cache performance. Release: mmftpd 0.0.15 devl Date : January 9, 2003 By : Matthew Mondor * Important change - When desired, it is important for frontend scripts to be able to generate required password hashes. This has not been easy with the previous method, which did not even use standard base64 encoding but a custom one on the MD5 result. Calling external binaries such as mmpasswd is not ideal. Moreover, I wanted to get rid of the mhash library dependancy. The solution was simple: password hashes now consist of standard crypt(3) generated strings. Both MD5 and DES modes are supported. However, mmpasswd will only generate MD5 ones with crypt(3). This also allows administrators to easily migrate from real system users, as this is the way system user password hashes are stored. An attempt was made to provide a utility to easily convert old hashes to new ones; This however is impossible due to a bug in the previous method which caused the last few bytes of the MD5 results to be lost during the base64 conversion. A recommendation is to generate random passwords for the users, notify them via email and then activate the new version using the new passwords say, a week afterwards. This password format will then remain unchanged for next versions. * Bugfixes - When DISPLAY_FILE was set, the file in the / directory would not be displayed at login, only when a CWD was made to it. Fixed. - treesize() and treesize_edit() now use off_t type which corresponds best to a system's type size for filesystem related offsets/sizes. On BSD systems this becomes 64-bit. - If an invalid path was used for CWD the server would not output the expected 550 error line and the client would freeze while waiting for the reply. This bug was introduced when path sanity checking was rewritten for exists() in mmftpd 0.0.12 devl. * Real asynchroneous functions support - The Pth library has limits in that only one process is used for all threads. The various pth_*() functions, and special care has to be taken to prevent locking the whole process when a single thread needs to perform operations, because of the non-preemptive nature of Pth. Another side effect of using Pth is that SMP systems gain no performance over single CPU systems. Moreover, some functions which can take a while can lock the whole process when no pth_*() wrapper function exists to do it. An obvious example is hostname resolving, which can lock the whole process for long periods on slow networks. - A solution was worked out to allow threads to execute real asynchroneous functions without blocking the main process (and therefore allowing other threads to remain responsive), and to even take advantage of multi-processor systems where available. - The technique uses an AmigaOS-like device task/thread which works over the Pth thread-safe message passing mechanism, to serve the various asynchroneous functions like a daemon would. This device internally uses a pool of real asynchroneous processes to which it dispatches the requests in a distributed manner via unix datagram sockets. - The new ASYNC_PROCESSES configuration file option was added as a result, which allows to specify the number of slave asynchroneous processes to run. - This facility is currently used by mmftpd to generate password hashes, to resolve hostnames (when enabled), to evaluate user home directory tree size (for accounts with quota limits), and to obtain a line corresponding to a user from mmftpdpasswd. * Miscelaneous new features - It is now possible to set "*" for the password of various users to allow password-less logins. These will accept any given password or empty ones. - mmstatd can now be specified configuration file to use at startup via command-line argument. - mmstat library will now first check for MMSTATCONF environment variable for the configuration file to use instead of the default "/etc/mmstatd.conf". This can be useful to non-privileged users who want to run mmstatd. - If the daemon is compiled with -DNODETACH, the main process will not fork(). Useful for debugging. - If compiled with -DNODROPPRIVS, daemon will make sure that it is not started by the superuser to accept running, but will then not attempt to perform any modifications on the current permissions. Useful for non-privileged users. The default is to only accept to be started by the superuser and then drop privileges. - mmstat user client utility now supports hreport for more human readable results (although less verbose). - Configuration file option DELAY_ON_ERROR was added to allow to pause at every user command error. Thanks to Jeroen Oostendorp for the idea. - mmstat facility now allows wildcard pattern matching for UPDATE, RESET and DELETE operations which operates atomically on all matching keys for that uid. * Other - Optimized the command matching loop by using fast packed hashes - Other optimizations were performed by moving some variables in closer scope, usually resulting in compilers using registers for appropriate variables. This is especially useful for GCC which ignores register directives. - Significant code cleanups - Several environment variables can now be set to modify the behavior of the install.sh script (eg: change installation prefix, default group/user, etc). - The memory pool allocation system was optimized even more, now keeping statistics on the average number of pages in use, so that it scales better. Previously used pages are now privileged as well, which causes unix kernels to allow processes using it to perform more efficiently, as well as the underlaying allocator. - Better synchronization with mmstat service, now using persistant keys and has possibility to delete old temporary keys at startup using wildcard matching. Release: mmftpd 0.0.14 devl Date : November 8, 2002 By : Matthew Mondor * New features - Added new CHROOT_DIR configuration parameter to optionally allow the server to run enclosed into a chroot(2) jail. - Added PASSWD_FILE configuration parameter to allow the administrator to override the name of the /etc/mmftpdpasswd file by a custom one. - When starting the server it is now possible to specify the configuration file to read rather than /etc/mmftpd.conf. This allows to start several copies using different configurations with the new CHROOT_DIR option. * Bug fixes - When multiple groups feature was introduced in 0.0.10 devl, mmftpd stopped setting the initial group using setgid(2), assuming that setgroups(2) would. This was not the case, and mmftpd would then remain part of the wheel group. It will now set the real and effective primary group, the secondary groups, and then the real and effective user. The first group of the GROUPS configuration file parameter is used for the primary group. Although the virtual chroot jail is safe, now that it is possible to disable file ownership checking for some accounts, and possibly use read-only accounts in wider areas when wanted, this needed to be fixed. - Removed the annoying LOOP: debugging which used to flood syslog. * Other - Now uses getnameinfo(3) instead of gethostbyaddr(3) to resolve hostnames if RESOLVE_HOSTS is TRUE. This is more suitable with threads, since gethostbyaddr(3) uses static data and required a mutex to be thread-safe. Release: mmftpd 0.0.13 devl Date : October 28, 2002 By : Matthew Mondor * Important bug fix - omission of an unlinknode() in the new rate limit feature was fixed. The daemons could lock in a loop taking 100% CPU time. Fixed. Release: mmftpd 0.0.12 devl Date : October 26, 2002 By : Matthew Mondor * New features - Anti-DoS connection rate limit feature was added in mmserver library, consequently new CONNECTION_RATE and CONNECTION_PERIOD configuration file options were added. These are on a per-IP address basis. - A new /etc/mmftpdpasswd (see mmftpdpasswd(5) man page) column has been added which can be used to enable/disable checking for file ownership. This sanity checking used to be obligatory, but it may be useful to disable it for some read-only accounts, for instance. - LIST and NLST will now properly evaluate each file for ownership and regularity before displaying it, no longer subject to glibc issue about it's incomplete opendir(3) implementation. In the case where ownership verification is performed for an account, files owned by another user than mmftpd will silently remain hidden from the user, but will still be logged via the syslog facility for the administrator. - Will now accept to resume really large files on systems which allow it. - The bytes statistic counters were bumped to 64-bit. - Globbing/pattern matching now uses fnmatch(3) although only operating on the last path element as it used to. As a result more complex patterns are allowed than '*' and '?' matching. At my great satisfaction, I can now host my netbsd package repository using mmftpd. * Bug fixes - Yet again a build/install script bug; Permissions would not be applied properly to newly installed files, the problem started to occur when automatic directory creation was added. This was now fixed. Users should consider upgrading as fast as possible, or to make sure that their configuration files in /etc/ have safe permissions. See the following man page: mmftpdpasswd(5). The install script will force safe permissions on the configuration files while preserving them if they already exist. - The staff group would not be created by install.sh if it did not exist. Thanks to Jeroen Oostendorp for reporting this. - mmpath(3) library's exists() function used to return the same 0 result if the file/directory did not exist or if it existed but was owned by another user than the one mmftpd ran under. This, under very particular circumstances could have lead to unexpected behavior. exists() now returns specific return codes for each situation, which mmftpd evaluates. Example of a situation in which this could have been harmful: STOU needs to make sure that a file does not exist before creating it. If a file owned by another user existed, and was writable by the mmftpd user, it would have been overwritten. A similar situation could have existed for RNFR/RNTO, where a file could have been copied/moved over another one. Of course a warning would still be issued in the logs. Normally, this never happened as files owned by another user (if any) usually were not writeable by others. But who knows, let's not assume anything. See the mmpath(3) man page for the new exists() semantics. - If a very long filename existed (of MMPATH_MAX length for instance) LIST would not be able to display the whole filename. Fixed. - The bandwidth shaping system was not accurate enough, as it used to throttle by 4096 bytes blocks, requireing too much CPU time in syscalls for high limits. It was rewritten using a new design which now only sleeps if required after the maximum number of bytes allowed into a second were transfered. So it now uses a second's worth of data resolution rather than a 4096 bytes resolution. Some parts of mmfd library internals and API were modified as a result (See mmfd(3) man page for details). Another side effect is that it takes far less CPU time with high limits. Thanks to Darren Price for the extensive testing and reports, to Eric Weisenhaus and agrajag for reporting these earlier. - There existed a potential race condition when hostname resolving was enabled, which could cause a memory leak. This was fixed. Thanks to Jeroen Oostendorp for reporting it. * Other - The mmpath(3) exists() and ls() functions were optimized to achieve better speed. - Binary file transfers are no longer using the buffered fdbread() and fdbwrite() functions but rather fdbrread() and fdbrwrite() directly, saving some CPU cycles and avoiding buffer size splits. - Some source code auditing and cleaning was made, several optimizations were performed, some buffers were aligned to favorize word-copy/move operations as opposed to single-byte moves, in mmstring library). - Finally this ftpd starts to be decent. - I thank everyone who are testing this software, and am again requesting that it be evaluated, as I intend to eventually release mmftpd v1 stable. For this to happen it has to be free of any issues, it is therefore important to report any bug, to mmondor@gobot.ca. It also would be nice if I could release with it a list of systems it is known to work on, so if betatesters find some time to describe their setup to me via email it would be appreciated. I yet have to test it again on ultrasparc with Linux and Solaris for instance. Release: mmftpd 0.0.11 devl Date : October 11, 2002 By : Matthew Mondor * Bug fixes - Although mmftpdpasswd(5) man page specified that the mfs column was expressed in bytes, the default etc/mmftpdpasswd and mmftpd daemon still considered it to be in kilobytes. This was now fixed to really use bytes for that entry. A block size could be 512 bytes, for instance. - The build scripts did not behave as expected on systems which are using bash for /bin/sh. Some script sections needed modifications to work on both. Now seems to build fine on linux systems. - Various new GCC versions warning issues were fixed. - When installing, some directories were assumed to exist, they will now be created when missing. - A bug in mm_memmov() was fixed. Release: mmftpd 0.0.10 devl Date : October 6, 2002 By : Matthew Mondor * New features - Now uses mmstatd(8) for persistant statistics storage and volatile who database. This includes mmstat(8) administration facility. This daemon also comports crash recovery using internal logging. See mmstatd(8) man page for more information. - A new column was added to mmftpdpasswd(5) to optionally allow download statistics counters on files available under a specific user. This is currently being used to record statistics of mmsoftware, files are stored under mmftpd, and the HTTP frontend links to files with FTP URLs under the mmsoftware user. - Daemon can now be part of more than one group, which is a requirement to access mmstatd. - It is now possible to set global maximum download and upload limits for the whole server. - Improved installation/upgrade make.sh and install.sh scripts, and use of /bin/sh rather than make because of GNU/BSD make inconsistencies. - mmftpd(8), mmftpd.conf(5), mmftpdpasswd(5), mmpasswd(8), mmstatd(8), mmstat(8), mmstat(3), mmstatd.conf(5), mmpath(3) man pages were written. * Bug fixes - On several systems, if mmftpd's group did not match with a virtual user's group new files would not be created with the right permissions. This was fixed by the new feature where mmftpd process can be part of several groups. - Spaces were not allowed in passwords, and empty passwords were not allowed for anonymous logins. This was fixed. - Since mmpath library bugfix that was made in mmftpd 0.0.7, pattern matching stopped working. valid_path() was now completely re-written around a cleaner design and is more solid. Moreover, multiple consecutive '/' in pathnames are now accepted and considered as a single '/', conforming to IEEE 1003.1. - On various systems, including OpenBSD, the NetBSD-style __RCSID and __COPYRIGHT, although useful, caused problems. They were now replaced by custom macros, which should fix compiling issues. Thanks to Dinos Costanti for reporting the issue. * Other - The code used to be compatible with older C compilers, it now was converted to ANSI C. Use of __P() macro is no longer made. - Some mmlist node manipulation functions were replaced by macros. mmlist(3) now privileges pre-allocated nodes when inserting them back in the free nodes queue, making a better useage of memory. Release: mmftpd 0.0.9 devl Date : July 22, 2002 By : Matthew Mondor * Bug fixes - Fixed bandwidth shaping library, recent versions would totally ignore bandwidth rate specified, because of erroneous zero initialization. Also fixed a bug where bandwidth was always lower than specified speed. Thanks to Eric Weisenhaus, Darren Price and agrajag for reporting these. - When clients send ABOR, they often send control sequences which resulted the command to not be recognized, this was fixed by adding a sevenbit boolean parameter to fdbgets() function. - Configuration value LOG_LEVEL would not accept values below 1 - Fixed strnicmp() which would sometimes cause various problems Release: mmftpd 0.0.8 devl Date : June 4, 2002 By : Matthew Mondor * Improvements - INPUT_TIMEOUT has been replaced by CONTROL_TIMEOUT and DATA_TIMEOUT so it is now possible to specify independant transfer i/o timeout. - Various optimizations * Bug fixes - A pretty serious bug was fixed (which only affected glibc-based systems), where syslog() would potentially be called with user supplied parts, including fmt sequences. Thanks to Benoît Roussel for providing me with a very detailed security advisory and to Guillaume Pelat for discovering the problem. Release: mmftpd 0.0.7 devl Date : May 19, 2002 By : Matthew Mondor * Config file support - Server now reads /etc/mmftpd.conf which obsoletes previous mmversion.h pre-compilation issue. - Defaults are set, configuration file is parsed, then command line arguments parsed which may override some of the configuration file settings. - Now possible to make install without overwriting the configuration file. - Syslog facility can now be specified. * Written script to ease compiling and upgrading faster. * Introduced a 64-bit hash function - Internally used to improve various table lookups (last visited directories and logged in users tables) - Hash function was tested against duplicates with /usr/share/dict/words and find / inputs with success * More verbose optional logging - When level 3 logging verbosity is used, replies are now logged as well, this is especially useful for beta-testers and debugging. * Added various motd-like features - WELCOME_FILE displayed at user connection if specified - MOTD_FILE displayed at successful login if specified - DISPLAY_FILE if specified will cause CWD into a directory to check for specified file name, and display it, if present. * Bug fixes - The mmpath library contained a bug which could crash the daemon, inapropriately using snprintf(), of which return values should not be trusted. This did not permit remote execution of arbritrary code from the client however, since mmfd's fdbgets() does not permit to read binary data. - Harmless, but still a bug, once into another CWD than /, the user could indefinitely CWD . which would cause PWD to become for instance something like this: /tmp/////// (upto MMPATH_MAX bytes long). Obviously any file command would thereafter fail with a 5xx error until CWD was changed. - The function which strips possible - parameters in commands used to prevent using filenames containing -. It now only strips options before any other parameters, as is standard on BSD systems. Thanks to Lior Marantenboim for reporting this bug. * Server too busy message introduced - When the maximum number of IP addresses have been reached, or when too many connections from the same address occurs, a protocol-friendly message is now sent before closing the connections. Some clients have been confused with previous behavior. * Multiple interfaces support - Adminstrator may now specify several IP addresses to bind() to, as well as a server hostname for each, using LISTEN_IPS and SERVER_NAMES config file parameters. * Now using getpwnam() and getgrnam() for more configurability - Used to read /etc/passwd and /etc/group directly. Release: mmftpd 0.0.6 devl Date : April 22, 2002 By : Matthew Mondor * Fixed a nasty bug - As the mmfd library was lately fully redesigned some daemon code would not properly use the status results when a connection was unexpectedly lost. - This rendered easy to crash the daemon flooding with alot of connections not exiting using standard QUIT command. * Better disconnection logging - The the general status/reason of the disconnection is now also logged with the statistics via syslog. * Compilation issues resolved - Fixed a conflict which occurred between the BSD and linux usleep() function not using the same argument type (unsigned long vs useconds_t) - mmpasswd would not compile properly on linux Release: mmftpd 0.0.5 devl Date : April 17, 2002 By : Matthew Mondor * Implemented bandwidth shaping support - mmfd library was modified alot, it now supports bandwidth shaping capabilities which mmftpd finally takes adventage of. - Two new fields were added in the mmftpdpasswd user configuration file for download and upload speed limits on a per-user basis. - A new mmversion.h global parameter was added to set read and write allowed speeds on the control connection. - Speeds are specified in KB/s and values as low as 1 will work, but 0 disables bandwidth shaping allowing maximum available transfer speeds. * Implemeted transfer statistics - Now records the number of bytes in and out of control and data connections, as well as number of files transfers in both directions. - Will show statistics at STAT and via syslog at logout. * More descriptive logging on DoS attempts - When a connection is rejected because the maximum number of IP addresses is reached, or that maximum number of connections for an IP address is exceeded, mmserver library now reports the offending IP address and reason via syslog. Release: mmftpd 0.0.4 devl Date : March 24, 2002 By : Matthew Mondor * Now unified in a CVS tree with various other daemons and main libraries - This allows a better development method, preventing inconsistencies and thus eliminating some bug sources, ideal for security oriented software, and following the general BSD way of coding. * All code now distributed under the BSD license. - I almost started to despise glibc already, and am coding almost exclusively on NetBSD lately. Moreover this license tends to be less restrictive. Almost all very useful glibc and linux features were BSD derived anyways, working fine since quite a while. - I however still intend to keep the code portable to linux * New buffering mmfd library - All I/O operations have been greatly improoved * Will now be faster to restart - TIME_WAIT lingering sockets would previously prevent mmftpd from starting until it timeouts, fixed. Release: mmftpd 0.0.3 devl Date : Mars 8, 2002 By : Matthew Mondor * Works better with some broken clients - Some clients send - options to various FTP commands, notably LIST, those will now be ignored instead of returning an error message. - More efficient and secure syslog calling on systems supporting vsyslog() (mostly BSDs as glibc doesn't have it) Release: mmftpd 0.0.2 devl Date : January 6, 2002 By : Matthew Mondor * Wrote new linked list library - More efficient node pre-buffering technique with allocnode()/freenode() only calling malloc()/free() occasionally - All of mmftpd core now uses those for efficiency - Added a few new options in mmversion.h configuration file, related to the page size that should be used. These values can be multiplied on servers with high load. * Implemented maximum login limit per user - A new column was added in the config file * Home directory size limit support - Two new columns in config file, for optional maximum limit, and minimum file size for filesystem - User shared structure remembering current home directory limit, so that even if there are simultanious logins of the same user quota is respected - Tree size is only re-evaluated from scatch if no current logins of that same user exists * Updated the README file - Added more details, fixed a few mistakes Release: mmftpd 0.0.1 devl Date : December 17, 2001 By : Matthew Mondor * Initial development release