En:HOWTO: Set Up A Community Repository
Contents |
Why?
If you want different USE flags on packages or you want that new version you've just found in an overlay or just want to play with something cool then setting up a commnity repo is for you. It allows you to compile portage ebuilds into entropy packages and upload them up to a ftp site.
More information can be found on the En:Community Repositories Page.
Basic Requirements
Here is a list of the requisites you need to satisfy before starting your own repository:
- Portage or any Portage-compatible Source Package Manager (to compile packages)
- Entropy >= 0.99 including sys-apps/entropy-server package.
- a remote FTP where uploading packages and their repository database
- HTTP or anonymous FTP to allow client downloads
- A basic Portage terminology knowledge
Setting up your own repository
server.conf
The first stage is setting up your server.conf file located in /etc/entropy.
Let's copy it and start editing
# cp /etc/entropy/server.conf.example /etc/entropy/server.conf
We have only a few parameters to set in this file.
Branches
The first thing to set up is which branch your repository will operate on, this will normally be a number that will relate to the version of sabayon you are building packages on/for. Usually you will only need to specify the primary number, ie 4.1 would be branch "4" and 5.2 would be branch "5".
branches|5
(Please note, one of the branch string listed above must be also set in /etc/entropy/repositories.conf)
Officialserverrepositoryid
The officalserverrepositopryid is the shortened name of your repository, it needs to be a ascii string and can use dots "." and hyphens "-" but not spaces, it is used as the "repository identifier". The Community Repository interface must be aware of what is the default repository, especially when you have more than one.
## this is just an ascii string identifier explained later officialserverrepositoryid|myrepository
or
officialserverrepositoryid|my-repo.ftw
Repository
This is the main configuration string of for your repository it details:
- Repository Identifier (see above) - This is the Officialserverrepositoryid that you put above
- Repository Description - A more verbose description of your Repository.
- Upload URI(s) - ftp:// paths to your FTP storage in the format "username:password@host", there can be multiple URIs to upload to multiple locations, its recommended to use rsync from one primary server as it will be significantly faster.
- Download URI - The publicly accessable address of your packages, this myst be a HTTP URI pointing to your repos root directory.
- handlers URL - Handlers are small optional server-side PHP script to ease the work of the interface, the handler parameter must be a HTTP URI pointing to the repository root directory.
This needs to go in a pipe seperated line such as below: See the example below:
## repository|Officialserverrepositoryid|<repository description>|<FTP upload/download URI 1> <FTP upload/download URI 2>|handlers URL (path to repositories root directory) ## we will write repository|myrepository|My very first repository|ftp://username:password@sabayonlinux.org/myrepositories/%7Chttp://www.sabayonlinux.org/myrepositories
Product
There is another critical parameter listed in /etc/entropy/repositories.conf called "product", at the moment, Sabayon has only one supported product, called "standard". So, be sure to have this set in this file, since we're going to make a repository for Sabayon Linux, product: standard.
=Initialising the Repository
If all is configured properly, you will find that running this command:
# equo community repo status
Will bring the following output
# >> @@ Entropy Server Mirrors Interface loaded: >> mirror: ftp://myrepository:xxxxxxxx@sabayonlinux.org/ >> @@ Entropy Server Interface Instance on repository: myrepository (type: community repository) >> @@ Currently configured repositories: >> __system__ >> myrepository >> !!! [repo:myrepository] Your default repository is not initialized. Do you want to initialize your default repository ? [Yes/No]
As you can see Entropy found that your shiny new repository is empty and asked you to initialize it. Do it!
It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.
Anyway, eventually it will finish and you should see somthing along the lines of the below:
>> !!! Entropy database is probably empty. If you don't agree with what I'm saying, then it's probably corrupted! I won't stop you here btw... >> !!! Entropy database: fixing KDE dep strings on etpdb:myrepository. Please wait ... >> * Mirrors have not been unlocked. Remember to sync them.
If you do see the above then your repoistory has been setup successfully! In the words of Joost_op "go get yourself a beer".
Compiling and Adding Packages
Overview
The next step (the fun bit) is filling it with some packages.
Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:
# equo database spmsync
Compilation
Now to add a package, its Compile time! The first step is to compile something using any Portage-compabile source package manager (Portage itself, Paludis, Pkgcore - anything that writes /var/db/pkg properly). Let's say we want to compile our own media-sound/amarok perhaps with "postgres" support or with different CFLAGS. Then, we spawn Portage:
# emerge -av media-sound/amarok
Note, if you get Equo called by Portage, you need to select "no". This because of a Portage behavioral change in the latest release (apparently it pulls into its xpak metadata equo's bashrc).
Adding the Package to Your Repository
Single Repository
Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:
# equo community repos update
You will be asked to push the newly compiled packages into your default repository. This is the output you should expect:
>> @@ These are the packages that would be added/updated to the add list: >> media-sound/amarok-1.4.9.1 >> Would you like to package them now ? (inside mynewrepo) [Yes/No]
Of course yes! We have now pushed media-sound/amarok into our local repository database!
Multiple Repositories
If you have more than one community repository, a quick way to switch the default (working) repository is using ETP_REPO environmental variable:
# ETP_REPO=myotherrepository equo community repos update
You can also export it, and it will persitant for that session.
# export ETP_REPO=myotherrepository # equo community repos update
You will see the switch right after having pressed the Enter key:
# >> @@ Entropy Server Mirrors Interface loaded: >> mirror: ftp://mynewrepo:xxxxxxxx@community.sabayonlinux.org/ >> @@ Entropy Server Interface Instance on repository: myotherrepository (type: community repository) >> @@ Currently configured repositories: >> __system__ >> mynewrepo >> myotherrepository
When uploading you can can add the --syncall argument to sync all the repositories in one go.
Uploading the Packages
We now need to upload the new binary package(s) and the updated repository database.
equo community mirrors sync
This is the output you should expect:
>> @@ Entropy Server Mirrors Interface loaded: >> mirror: ftp://mynewrepo:xxxxxxxx@sabayonlinux.org/ >> @@ Entropy Server Interface Instance on repository: mynewrepo (type: community repository) >> @@ Currently configured repositories: >> __system__ >> mynewrepo >> * Starting to sync data across mirrors (packages/database) ... >> @@ [repo:mynewrepo|sync|branch:3.5] packages sync: sabayonlinux.org >> @@ Local statistics: >> @@ upload directory: 1 files ready >> @@ packages directory: 0 files ready >> @@ Remote statistics for: sabayonlinux.org >> * [ftp:sabayonlinux.org] connecting with user: mynewrepo >> * [ftp:sabayonlinux.org] switching to: / >> * [ftp:sabayonlinux.org] switching to: / >> * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/ >> * [ftp:sabayonlinux.org] switching to: 3.5 >> @@ remote packages: 0 files stored >> @@ Calculating queues ... >> ** Expanding queues: >> [branch:3.5|upload] media-sound:amarok-1.4.9.1~0.tbz2 [182.9kB] >> @@ Packages to be removed: 0 >> @@ Packages to be moved locally: 0 >> @@ Packages to be uploaded: 1 >> @@ Total removal size: 0.0b >> @@ Total upload size: 182.9kB >> @@ Total download size: 0.0b Would you like to run the steps above ? [Yes/No]
Last chance to double check what is getting done. We will choose Yes.
>> @@ [sabayonlinux.org|upload] connecting to mirror... >> @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/... >> * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/3.5 >> @@ [sabayonlinux.org|#1|(1/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2.md5 >> @@ [sabayonlinux.org|#1|(1/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2.md5: not supported >> @@ [sabayonlinux.org|#1|(1/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2.md5 >> @@ [sabayonlinux.org|#1|(2/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2 >> @@ [sabayonlinux.org|#1|(2/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2: not supported >> @@ [sabayonlinux.org|#1|(2/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2 >> @@ [branch:5] upload completed successfully: sabayonlinux.org >> Please insert a commit message: my first commit >> * Remote Entropy Database Repository Status: >> Host: sabayonlinux.org >> * Database revision: 0 >> * Database local revision currently at: 1 >> @@ [repo:mynewrepo|sabayonlinux.org] mirror successfully locked for download >> * [repo:mynewrepo|sabayonlinux.org|upload] preparing to upload database to mirror >> * Entropy database is already locked by you :-) >> * Mirrors have not been unlocked. Remember to sync them. >> * Entropy database is already locked by you :-) >> * Mirrors have not been unlocked. Remember to sync them. >> * [repo:mynewrepo|sabayonlinux.org|EAPI:3] preparing uncompressed database for the upload >> database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db >> * [repo:mynewrepo|sabayonlinux.org|EAPI:2] creating compressed database dump + checksum >> database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db >> dump: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2 >> dump checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2.md5 >> opener: bz2.BZ2File >> Database Export completed. >> database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db >> compressed database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.bz2 >> compressed checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.md5 >> opener: bz2.BZ2File >> @@ [sabayonlinux.org|upload] connecting to mirror... >> * [ftp:sabayonlinux.org] connecting with user: mynewrepo >> * [ftp:sabayonlinux.org] switching to: / >> * [ftp:sabayonlinux.org] switching to: / >> @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/... >> * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/database/amd64/ >> @@ [sabayonlinux.org|#1|(1/14)] uploading: make.conf >> @@ [sabayonlinux.org|#1|(1/14)] upload successful: make.conf >> @@ [sabayonlinux.org|#1|(2/14)] uploading: packages.db.bz2 >> @@ [sabayonlinux.org|#1|(2/14)] upload successful: packages.db.bz2 >> @@ [sabayonlinux.org|#1|(3/14)] uploading: packages.db >> @@ [sabayonlinux.org|#1|(3/14)] upload successful: packages.db >> @@ [sabayonlinux.org|#1|(4/14)] uploading: packages.db.dump.bz2.md5 >> @@ [sabayonlinux.org|#1|(4/14)] upload successful: packages.db.dump.bz2.md5 >> @@ [sabayonlinux.org|#1|(5/14)] uploading: packages.db.revision >> @@ [sabayonlinux.org|#1|(5/14)] upload successful: packages.db.revision >> @@ [sabayonlinux.org|#1|(6/14)] uploading: packages.rss >> @@ [sabayonlinux.org|#1|(6/14)] upload successful: packages.rss >> @@ [sabayonlinux.org|#1|(7/14)] uploading: updates.rss >> @@ [sabayonlinux.org|#1|(7/14)] upload successful: updates.rss >> @@ [sabayonlinux.org|#1|(8/14)] uploading: packages.db.dump.bz2 >> @@ [sabayonlinux.org|#1|(8/14)] upload successful: packages.db.dump.bz2 >> @@ [sabayonlinux.org|#1|(9/14)] uploading: package.mask >> @@ [sabayonlinux.org|#1|(9/14)] upload successful: package.mask >> @@ [sabayonlinux.org|#1|(10/14)] uploading: package.keywords >> @@ [sabayonlinux.org|#1|(10/14)] upload successful: package.keywords >> @@ [sabayonlinux.org|#1|(11/14)] uploading: packages.db.md5 >> @@ [sabayonlinux.org|#1|(11/14)] upload successful: packages.db.md5 >> @@ [sabayonlinux.org|#1|(12/14)] uploading: package.unmask >> @@ [sabayonlinux.org|#1|(12/14)] upload successful: package.unmask >> @@ [sabayonlinux.org|#1|(13/14)] uploading: profile.link >> @@ [sabayonlinux.org|#1|(13/14)] upload successful: profile.link >> @@ [sabayonlinux.org|#1|(14/14)] uploading: package.use >> @@ [sabayonlinux.org|#1|(14/14)] upload successful: package.use >> * [repo:mynewrepo|sabayonlinux.org] mirror successfully unlocked >> * [repo:mynewrepo|sync] database sync completed successfully Should I continue with the tidy procedure ? [Yes/No] >> @@ [repo:mynewrepo|tidy|branches:5] collecting expired packages >> @@ [branch:5] collecting expired packages in the selected branches >> @@ [branch:5] nothing to remove on this branch
Done and Done!
