Delphi/Perl Ftp Servers

*** TIME FRAME 30-45 DAYS *** I need a "simple" ftp relay system. There will be two main components, described in detail below, working together in passive transfer mode. FTP server one will connect to FTP server two and transfer files to the later in chunks. The co-worker should be familiar with Pascal, Perl, RFC-959 FTP protocol [login to view URL] and have some knowledge in cryptology. Booth server components should understand the basic standard FTP commands and reply in a correct manner. The programs must be coded to make it easy to understand, using a module schema with subroutines and functions where appropriate. Full payment will not be paid until the project is 100% done, source codes are delivered and the programs tested proven to be working correctly together. The parties agree that the source codes, complete and partial, together with the compiled builds will be the buyer’s property alone. Excellence, fast execution and pride of work are rewarded. FTP SERVER ONE -------------- Written with Delphi or FreePascal and executable on Windows. The program should be "invisible", and have a option to run only once (send a batch of files automatically to server two then quit) - normal start will put the program in sleep. Waiting for an incoming connection using as little system resources as possible. On Windows XP the program should preferable be able to run as a "service" in the background. On a connection and ip is allowed (otherwise connection should answer with 421 Access denied) it should answer with a configurable 220 message. If localcopy option in enabled the program will wait for the USR command which in our case describes the data source. The user string (eq c: on A windows machine) is then checked if existing and the "volume name" from this path will also be noticed. If accurate we will now connect to server two with handshaking (see appendix validation), otherwise we will answer again with the 421 message. If previous source path exists SITE SOURCE "$source" "$volume name" is sent once connected. Now the incoming FTP commands and data (described in next section) will be "relayed" to server two. The directory commands CDUP CWD PWD LIST NLIST will get it's content from server two, but if localcopy option is enabled we will also change current local directory for later retrieving more specific file information (SITE ATTR later described). Some basic commands such as SYST, TYPE, RETR (refused at this moment), PASV, NOOP will be handled locally together with answers to not supported commands. FEAT (RFC 2389) will list the MDTM command with the ability to set and read file time stamps which shall be implicated. If server one gets a QUIT command it will also disconnect from server two. When receiving a new STOR command we will send a SITE DEST "$filname" command - which shall tell server two that we are beginning with a new file. Then we will read and split the incoming data in chunks of 60.000 bytes (configurable), or less if we are at the end of a file or the file is smaller then this size). Now we will compress (see appendix fast compress) the data and thereafter shall a SHA-256 hash value with 64 bytes [login to view URL] be calculated and sent to server two with the command SITE CHUNK $string. We will then receive one of two possible messages from server two, indicating wherever the chunk is already in the storage or not. If not in storage we will proceed with encryption (appendix crypto method) and finally send it to server two using standard ftp commands such as PASV and STOR. If a new chunk (unique hash) is sent, server two shall respond after transmission with the correct hash value which again is calculated after the chunk is decrypted and stored (only if correct hash) on the other side. If transaction not successful we should try again sending the chunk. This process is then repeated until all chunks and file data are successfully transferred to server two. After a file is finished server one shall send a SITE ATTR with complete file attributes such as creation date, write-protected, hidden and so on - if available (localcopy=1). In backup mode 1 (default is 0) the chunk is first compressed then encrypted and finally the hash value is calculated and sent to server two. All filenames, directory names and paths should also be encrypted (if not crypt level is set to 0) when server one and two communicates in both ways - with a small improvement in the encryption scheme. Before this encryption a total of 0 to 256 random extra bytes will be inserted in the data string containing the actual name or path at random places. First byte tells that this special byte appearing anywhere in the name represent a random sequence. For example 02 [*?] 02 08 [*?] there 02 is the special byte and the following 08 bytes is just nonsense random bytes). Also the hash value then sent over tcp is encrypted - with standard scheme. (ini file etcetera, separated from code) 220 answering message system (SYST command) answer configurable port to listen on ip mask (which ip:s are allowed to use this server) localcopy (0 or 1) serveraddress/hostname or IP:port (ftp server two) username (-"-) cryptokey cryptolevel (0-512 bites) backupmode (0 or 1) privatecryptokey (if backup mode 1) privatecryptolevel (if backup mode 1) chunksize (60000 by default) list of files and or directories to be automatically transferred every execution FTP SERVER TWO -------------- Written in Perl and working under Linux environment. A good start to look is [login to view URL]~rwmj/Net-FTPServer-1.122/lib/Net/[login to view URL] (a free ftp server written in Perl). Using MySQL as database and storage engine - Perl interface and module is available at http://mysql.org. The program shall in some way accept and work with more then one connection at the same time. Server will wait for a connection on a configurable port. If ip exists in the field failedlogin sql table and the last login was less then 15 minutes ago is should immediately send a 120 (service ready in nn minutes) message followed by a 421 access denied response and close down the connection. If that's not the case the server shall answer with a 220 message and ask for user to login. The user name is checked in the sql users table, if exists we will proceed with the validation routine (see appendix validation). Login failure including timeout will result in logging the ip and timestamp in the failedlogin table, successful login will be saved in the users table. Next the program will wait for the SITE SOURCE "$source" command - stopping everything else with a 451 message "451 Waiting for SOURCE command, can not proceed". Once this command is received and the data strings containing the actual source strings is decrypted, the program shall look up the path in the directory table. Notice a sub source (eq. c:temp2) shall be found browsing from root start (eq. c: in database). If not found new post(s) shall be created in the table directory. Also - same source with different volume names should be saved in separate posts. A normal tree with one sub directory will include the following posts. 1 root post (rooid.directory=0, name.directory=$rootsourcename, nextlevel=$x, user.directory=$currentuser, servertimestamp.directory=$timestamp, attr.directory=$volymename) 1 root level post - with the subdirecoty, more posts with same $x value if more subdirectories or files on this level (rooid.directory=$x, name.directory=$directoryname, nextlevel=$y;entries in this directory, user.directory=$currentuser, timestamp.directory=$timestamp, servertimestamp.directory=$timestamp, attr.directory=$attr). Browsing with the CDUP and CWD command shall of course be possible. When server one asks for directory listings, file information or similar server two should send this, starting from the root post as described above. Notice only the latest file or directory version should be visible (checking the timestamp field), and any action such as file attribute changes or DTM should be applied on this version (for this session). If server one wants to delete an entry we should create a duplicate of that post, update the server time stamp field and mark it as deleted by inserting the text "*DEL*" in the chunks field. These two entries should of course not be visible under normal circumstances. If the special command SITE DEST has been sent we are ready to accept a new file. Thereafter we will create a new entry in the directory table, rootlevel field filled with the correct value from nextlevel field - depending on source path and previous directory commands (server one position in the "tree"). For more information about the transferring of chunks, see under server one details. Please note that we save a new entry for every file for every session, but only showing the latest copy version looking at the timestamp, and if deleted showing nothing. Every chunk id for a file is stored in chunks field in directory table, making it possible to rebuild any version of a file. A new chunk (not existing in the database) is decrypted if backup mode 0, but not if mode 1 - and the mode is saved in the backupmode field. If a sub directory is deleted and later created again (same name) with the MKD command it will always get a new nextlevel value to separate items in the directories. Finally the server program shall have an option to output and save any directory listing and file(s) from the storage in a decompressed format, maybe a separate perl script will suit best for this. APPENDIX - SQL TABLES ------------------------ failedlogin > ip > timestamp users > username > maxchunks (maximum chunks created by user allowed, server two should answer with 552 if exceeded) > maxfiles (maximum number of files in storage allowed, server two should answer with 552 if exceeded) > uniquechunksused (number of unique chunks created by user) > totalchunksused (number of total chunks associated by user files) > numberofbackupsfilescurrentlyinstorage (see max files) > cryptokey > cryptolevel (0-512 bites) > lastlogin (shall be updated after every successful login) > lastbackup (servertime) > numberofdays (increased by one whenever a new backup differ from last backup above) > userid chunks > hashvalue (in decrypted format) > data (compressed, but decrypted if created using backup mode 0 - otherwise encrypted) > numberofrepeatedusage (increases every time a this chunk is contained in a file) directory > rootid (always 0 on source root, otherwise increasing with a unique number for each new level created) > name (source, file or directory name) > nextlevel (if this is a directory, sub level can be reached with SELECT user = $user AND rootid = [login to view URL]) > user (owner connected to userid in the users table) > timestamp (timestamp received from server one with the FTP DTM command) > servertimestamp (our internal timestamp when this post was created, should be universal format in seconds not yymmddhhss) > attr (original attributes from server one, SITE ATTR) > backuptype (0 or 1) > shared (reserved field for future use) > chunks ("*DEL*" if deleted, if file hash values in a single row for every chunk of data within this) > totalsizeinbytes (files only, may differ from the number of bytes contained in the chunks, because of the compression) APPENDIX - FAST COMPRESS ------------------------ To save some bandwidth and avoiding certain patterns, each package with chunk data shall be compressed. Every byte in the data stream is compared with the previous and if 4 or more bytes are the same we will remove them from stream and instead add codes for them in a list which shall be appended at beginning of the stream of data. Op-code #00 marks end of list (data stream follow) #01 tells the number of repeats is 2 bytes word following this op-code, otherwise #04-FF tells the number of repetitions. Following byte is the one repeated and next two bytes tells the original placement where the repeated bytes was placed in the data stream, and so on. Here is a little example; 06 2A 00 02 00 65 01 03 4F - will translate to 65 01 2A 2A 2A 2A 2A 2A 03 4F. APPENDIX - VALIDATION --------------------------- Instead of a single password server one and two have to secure identification using some kind of a validation routine. At the same time server one has to tell server two which crypto level this session will use and if the files are considered type backup mode 0 or 1. The schema has to be cryptology safe i.o. not using recognisable strings or values in the decrypted form. This part I'm not totally sure how to design, but there are several examples on the net. My thought was using some kind of one time password. Server two begin with asking for user name, checked with the sql table users. Once accepted server two will choose a random password, reading x bytes from the stream generated by this seed and encrypt the one time password with our secret password. The encrypted data is showed as text in the 331 (User name OK, Password Required) message 331 Enter one time password. Server one decrypt the password and returning the bytes from the stream. Of course the response from server one is also encrypted. The problem is that this schema only works one way, how shall server one know that server two is genuine and how shall server one set the crypto level and tell the backup mode? This is up to the programmer accepting this project to solve. APPENDIX - CRYPTO METHOD ------------------------ The server software shall use a form of stream cipher, when transferring chunks, hash values, file or directory names (not commands!) as explained in detail earlier. The bytes from the stream is added or xored to the bytes to be encrypted. This is done with a secret key booth parties share, acting as seed, and a cryptographically secure pseudorandom number generator called ISAAC. Example code for both Pascal and Perl exists here [login to view URL] Also each package (maximum length equals configurable chunk size) shall be encrypted using a unique "message key", which is created in run time with a mix of current system time in micro seconds, programming language random function and ISAAC. The message key is then encrypted itself with the secret key and appended to the package. The secret key string (ini file in server one and in users table) shall be represented as a hexadecimal number of any length. This number is downsized to a length of 0 (turned off) to 512 bits, depending on current crypto level as specified in the ini-file.