<<< Want to ask a question - go to this forum >>>

Vision

Project Outline
Packages from the Debian-Distribution are obtained from public http-mirrors. To save bandwidth our program will function like a drop-in, stand-alone proxy for the internal network. But in addition it will store already requested packages locally. Packages that are not yet stored locally are fetched upon client-request from the parent server transparently.

Problem Statement
To write a program that improves the performance of the Linux Debian package update and install application named apt-get.

Key high-level goals
- Reduce bandwidth usage
- Reduce download time

System features
- Handle client request
- Respond to request from client
- Request files from remote server
- Handle server responses (e.g. file not found - 404, etc.)
- Cache files locally
- If multiple clients request a specific file simultaneously, download only once from remote server
- Handle disk usage (purge old files)

Other requirements and constraints
- Watch out for "full disk"
- Prevent "denial of service" attacks by limiting number of threads

That said...

I am writing this program because I looked at all the solutions for mirroring debian that are out there - and found that none of them is suitable for my purposes. Most of the other implementations are shell scripts and therefore very limited, when it comes to customization. And some of them are simply slooooow.

apt-got supports all architectures and will automatically adjust to your needs. Furthermore it is not limited to Debian archives. The SimpleMirrorModule can be used to mirror ANY filestructures. It is even very easy to implement own modules. (I focused on a very clean and intuitive module API). Future releases will include more sophisticated caching algorithms.

By starting with a real mirroring engine I hope to lay a foundation for further improvements. Right now I plan on these improvements:

Critical

• Create configuration infrastructure.
• Dynamically load modules, prepare for different kinds of module flavors
• Solve “blocking”-problem when file is not yet downloaded.
• Create package-lists parser.
• Create intelligent caching/purging algorithms.

Important

• Increase delivering speed to client by creating an apache module.
• Parse all available source-lists and merge them into one.
• Get MD5sum from source-lists and check downloaded data before storing.
• Create statistic-/download tracking-extensions.

Optional

• Create pre-fetch mechanism for frequently requested files and/or dependencies.
  

For a detailed description of these improvents procede to the proposal.