apt-got - a simple mirror engine

http://hacktor.fs.uni-bayreuth.de/apt-got/

0. Introduction

0.1 Content

0. Introduction
0.1 Content
0.2 Author
0.3 Latest version

1. Compiling
1.1 Compiling using ant
1.2 Compiling by hand

2. Configuring apt-got
2.1 Server part
2.2 Mirror Engine
2.2.1 Mirror Control part
2.2.2 Mirror Modules
2.2.3 Proxy Support

3. Running StandAloneAptGot

0.2 Author

This documentation of apt-got is written by
Tobias Hertkorn - <t.hertkorn@gmx.de>
© 2004

0.3 Latest version

The latest version of apt-got and this documentation can be picked at http://hacktor.fs.uni-bayreuth.de/apt-got/

1. Compiling

1.1 Compiling using ant

Go to the root repositiry of the source (where the build.xml is) and simply type 'ant'.
If your system is set up correctly this should start the building process and produce usable results in build/

1.2 Compiling by hand

If you can not use ant on your system (although it is very easy to set up on Debian systems) you can always compile apt-got by hand.
To do so change into the src subdirectory and type (or copy and paste) the following commands:

For UNIX/Linux:

javac *.java
javac com/debianmirror/http/*.java
javac com/debianmirror/mirror/control/*.java
javac com/debianmirror/mirror/data/*.java
javac com/debianmirror/mirror/module/*.java
javac com/debianmirror/server/*.java
cp -r * ../build/

For Windows:

javac *.java
javac com\debianmirror\http\*.java
javac com\debianmirror\mirror\control\*.java
javac com\debianmirror\mirror\data\*.java
javac com\debianmirror\mirror\module\*.java
javac com\debianmirror\server\*.java
copy /r * ..\build\

2. Configuring apt-got

2.1 Server part

StandAloneAptGot.props for server specific configs

# apt-got.server.listeningPort
# specifies the port our server will listen on for clientrequests
apt-got.server.listeningPort = 8085
# number of workers to start in the beginning.
# Will adjust dynamically 
apt-got.server.nrOfWorkers = 5
# Where is the Mirror configuration file stored
# Can be a relative path (to the directory where the server was started)
# or as an absolute path
apt-got.mirror.configHandlerExtraInfo = conf/mirror.xml
#
# you probably don't want to change anything below this line
apt-got.mirror.configHandler = com.debianmirror.mirror.data.XmlMirrorConf
apt-got.server.nameSpace = com.debianmirror

There are only 2 settings in here that are really interesting for you:

apt-got.server.listeningPort to set the port StandAloneAptGot is listening on
apt-got.mirror.configHandlerExtraInfo to set the path to the configuration file for the mirror engine. This can ether be a relative path to the directory you typed the java command in, or an absolute path.
Windows user will have to change this to conf\mirror.xml in order to get correct results.

2.2 Mirror Engine

2.2.1 Mirror Control part

As a default the mirror control configuration is stored in conf/mirror.xml.

<mirror>
  <control>
    <namespace>StandAloneAptGot</namespace>
    <logger>
      <namespace>com.debianmirror</namespace>
      <level>all</level>
      <filename>localhost_control_log.txt</filename>
    </logger>
    <moduleloader>
      <filemask>^module_.*\.xml$</filemask>
      <rootdir>conf/modules</rootdir>
    </moduleloader>
  </control>
</mirror>

Again there are only 2 setting that are interesting for you. In the <moduleloader> context the settings in <filemask> and <rootdir>.

<filemask> is a regex. The default tells the ModuleConfigLoader to look for filenames that start with module_ and end in .xml - module_debian.xml or module_simple.xml would match that pattern.
<rootdir> is the directory the ModuleConfigLoader will look to find the module configuration. This is ether relative to the path you executed the java command in - or an absolute path.

As a sidenote: the logger configuration is not working as of April 15th 2004. But you can turn off all logger information by changing com.debianmirror in <logger><namespace> to something else.

2.2.2 The Mirror Modules

For every server (or more specific every directorytree on the server) you want to mirror you have to set up a mirror module. Remember that this is done in the subdirectory specified in mirror.xml with a specific filename mask.
If you did not change anything in mirror.xml this directory is config/modules/ and the file must start with 'module_' and end with '.xml'.
A good practice is to use the path you want the content to show on your apt-got server as the middle of the filename. Eg. if you want to mirror debian and it should show up as /debian on your server you should name the file 'module_debian.xml'.

Let's walk through the configuration of a Debian mirror module that provides the same information as stored in http://ftp.uni-erlangen.de/pub/Linux/debian/ in the directory /debian of our StandAloneAptGot server and all mirrored data will be stored in the debian subdirectory of the directory we executed the server in:

Create a module configuration file named 'module_debian.xml'.

Insert following information in this file:

<mirror>
  <module>
    <parser>com.debianmirror.mirror.data.XmlDebianMirrorModuleConf</parser>
    <classname>com.debianmirror.mirror.module.DebianMirrorModule</classname>
    <path>/debian</path>
    <storage>
      <path>debian</path>
      <maxsize>2048</maxsize>
    </storage>
    <remote>
      <main>http://ftp.uni-erlangen.de/pub/Linux/debian</main>
      <backup>http://ftp.uni-erlangen.de/pub/Linux/debian</backup>
    </remote>
    <logger>
      <namespace>com.debianmirror</namespace>
    </logger>
  </module>
</mirror>

Here are the importent parts of the configuration:

<path> sets the path where this module shows up in the StandAloneAptGot.
E.g. /debian will translate to http://localhost:8085/debian
<storage> <path> sets the path where the cached files will be stored in.
E.g. /tmp/debian
<remote> <main> sets the URL of the remote server (Where to start mirroring from).
This should be the root of the remote Debian archive (That means the URLs as listed in http://www.debian.org/mirror/list). Both HTTP and FTP are supported, but I advise you to use HTTP server.
E.g. http://ftp.uni-erlangen.de/pub/Linux/debian

As of April 15th 2004 following directives are ignored:
<storage> <maxsize>
<remote> <backup>

2.2.3 Proxy Support

If you need to go through a proxy to contact your remote server, that is not a problem with Java. Please use the -D option when you start the StandAloneAptGot to set the appropriate system properties.

http.proxyHost (default: <none>)
http.proxyPort (default: 80 if http.proxyHost specified)
http.nonProxyHosts (default: <none>

ftp.proxyHost (default: <none>)
ftp.proxyPort (default: 80 if ftp.proxyHost specified)
ftp.nonProxyHosts (default: <none>)

http.proxyHost and http.proxyPort indicate the proxy server and port that the http protocol handler will use.

http.nonProxyHosts indicates the hosts which should be connected too directly and not through the proxy server. The value can be a list of hosts, each seperated by a |, and in addition a wildcard character (*) can be used for matching.
For example: -Dhttp.nonProxyHosts="*.foo.com|localhost".

ftp.proxyHost and ftp.proxyPort indicate the proxy server and port that the ftp protocol handler will use. ftp.nonProxyHosts is similiar to http.nonProxyHosts and indicates the hosts that should be connected too directly and not through the proxy server.

For more details go to:
http://java.sun.com/j2se/1.4.1/docs/guide/net/properties.html

3. Running StandAloneAptGot

Go to the root directory of the source repository (where the build.xml file is located). I assume you compiled the source and configured the mirror engine.

A convenient way to start StandAloneAptGot is to type in:
java -cp build StandAloneAptGot

Hint:
StandAloneAptGot knows some command line switches. Try java -cp build StandAloneAptGot --help

Appendix:

A. System requirements

Every system with an installed JDK 1.4 can compile and run the sourcecode.

B. Known problems

As of May 20th 2004

Download of Release and Packages files blocks until all data is recieved from the remote server (and md5sum are computed and compared)
Packages files are not refreshed automatically. Yet. =)
only first level subdirectorys supported as path in the mirror module configuration <path> directive.