Additionally, NewsPlex can be used to quickly explore large lists of candidate news-servers and evaluate their service value. This is called the Minimal mode.
NewsPlex is a process which runs on your own machine and appears like a standard news-server.
NewsPlex fully merges what the different news-servers offer. It creates news-groups which contain all available articles on all the servers.
NewsPlex can be seamlessly inserted between your existing news-server and your news-reader, without disturbing the list of groups or the article numbers.
NewsPlex does not retrieve articles before you request them. It only creates local indexes and redirects article retrieval as necessary.
NewsPlex always uses the best server for requesting an article, among those having it. It will use the remaining servers if the best one could not deliver the article. Servers are rated dynamically according to speed, latency and availability.
NewsPlex will close the connection to the client news-reader in case of errors appearing in the middle of an article retrieval, typically server timeouts. The NNTP protocol limitations do not allow the recovery of such situations. NewsPlex will however remember the error condition, and if the article is asked for again, it will try to use a different server for obtaining it.
NewsPlex can be run in the background, permanently. It will explore news-servers periodically to maintain its article database up to date.
NewsPlex does not need to be stopped if the network connection goes down, for example because you disconnect from your ISP. It will wait until the connection is up again and continue operations.
NewsPlex is able to use any news-server as news source, even news-servers which implement very old versions of the news transfer protocol, and yet always exports a modern interface to client news-readers. Errors occurring during the exploration of servers are also recovered, thru either restarts, deferring or workarounds.
NewsPlex supports connecting to news-servers either directly, or through proxy servers (SOCKS 5 or Sun-specific).
Currently, NewsPlex does not allow posting articles.
If your news-servers are unreliable, become easily unreachable or give timeouts, NewsPlex will isolate you from these errors, perform the necessary retries in the background and keep you posted with the latest news in the minimal time and without drudgery.
Few news-readers allow using several news-servers at the same time, and no news-reader known to me can take articles from several servers for a single news-group transparently, or request an article from different servers until successful. NewsPlex allows you to choose your news-reader on other criteria than these abilities.
NewsPlex version 1.4 is freeware. It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. You can use at your own risk, and report whatever problems you have encountered with it. You are free to copy this software ONLY if you include this document file and all the other accompanying files with it. You may NOT charge anyone for a copy of this software other than a small copying fee. You may NOT include this software with any commercial software without the written consent of the author.
Unpack the NewsPlex distribution archive to any directory. You should have one executable file newsplex.exe and one help file newsplex.htm in HTML format, which you are currently reading.
NewsPlex requires one subdirectory and two files in it to start. They must be located either in the directory from which NewsPlex is started or in the directory indicated thru the -d command-line option:
| |
|
| config/servers |
This text file contains the list of news-servers to use
and identifies the main one.
It can also specify which proxy, if any, should be
used for accessing a given server.
Each line describes a single server and has following format:
Eg: 1 news1.yourisp.com # server without login -2 mylogin:mypass@news2.yourisp.com # server with login (the main server) 3 news3.yourisp.com:4000 # server on port 4000 # this is a comment server-number must be unique for each server and greater than 0, except for the main server entry. It identifies the server in the article database and cannot be changed after a server has been explored. If the server-number is negative, the news-server will be considered as the main news-server. Only one news-server can be main. flags are undocumented for now; do not set them. It is possible to add comment lines, starting with a #, or to put comments at the end of lines, starting them with #. A line containing just the word exit will make NewsPlex ignore the rest of the file. A line containing just the word dontupdate makes NewsPlex not automatically update the servers which follow. By default, NewsPlex attemps to connect to servers directly. It is however possible to specify a proxy for each server. First, define a list of proxies you intend to use, like this: proxy number type url The number must be from 0 to 2. The type can either be none, or socks5, or socks5b, or sun. The url should specify the name of the proxy server, the port on it (default is 1080 for socks5 and socks5b servers and 4666 for Sun servers), as well as the login and the password if necessary. No url should be provided if the type is none. Note: socks5b servers are buggy socks5 servers. If you observe that NewsPlex is unable to connect at NNTP level to any server, and the logs/results file shows truncated welcome messages from them, with missing first 6 chars (eg. instead of 200 newsservername... you see wsservername..., and instead of 502 You are not allowed to talk you see u are not allowed to talk, use socks5b instead of socks5 to define the type of your proxy. For each server entry, it is possible to reference the proxy in the server-url by adding a final ,number to it. By default, value 0 is assumed. Example: proxy 0 socks5 login:pass@socksproxy:1080 proxy 1 none 1 news1.yourisp.com # server outside, will go thru proxy 2 mylogin:mypass@companyserver,1 # server on local network, no proxy 3 news3.yourisp.com:4000,0 # another outside server |
| config/groups.yes |
This file contains the names of news-groups which
should be retrieved from news-servers and offered to
client news-readers. The names of news-groups can contain *
and ? wildcards. Eg:
*midi* *music* Think twice before putting a sole * on a line into the config/groups.yes file, since this will make NewsPlex take all news-groups from all the servers, and create a really huge article database. |
You should select your existing news-server as the main server. NewsPlex will then synchronize with it when run for the first time, so as to provide the same groups and article numbers. This way, your news-reader will not be confused about which articles have already been read and which have not. NewsPlex will then complete the list of groups and articles with what other news-servers offer.
If you do not select any server as main, NewsPlex will consider all the servers as equivalent, explore them simultaneously, and number articles starting from 1 up.
Two further files are optional:
| |
|
|
config/groups.no
(optional) |
This file indicates the names of unwanted news-groups
which might get selected by config/groups.yes.
No wildcards are allowed here. Eg:
alt.music.boring alt.midi.lame |
|
config/groups.fix
(optional) |
This file indicates news-groups which should never
be filtered out by the config/groups.no file. This
is mostly a security against putting wrong content
into the config/groups.no file. Eg:
alt.music.best alt.midi.best Also, if a group vanishes from a server, this file allows to classify it as Wanted if it was listed there, Unwanted if it is now in config/groups.no, or Accepted if it was in neither of these files. |
If you modify any of these files, you will need to restart NewsPlex before the modifications are taken into account. The restart can be performed with the NPRESTART command or by shutting down the server and re-running it.
If you want to run NewsPlex without being connected to the Internet, you should execute the ifconfig lo 127.0.0.1 command first. Also make sure you have a TCPIP\ETC\hosts file in your OS/2 directory, or more generally a hosts file in the directory pointed by the ETC environement variable, and that if contains a line like 127.0.0.1 localhost, otherwise you will not be able to use localhost to reference your machine locally.
To shutdown NewsPlex when it runs in the foreground you can send it a Ctrl-C or a Ctrl-Break signal. A second signal will terminate NewsPlex unconditionally.
It is also possible to stop NewsPlex by telnetting into it (telnet localhost 119) and entering the NPSHUT command.
As the last recourse, you can terminate NewsPlex using a process killer like pspm2.
| |
|
| cache | The cache directory is created but presently unused. It is automatically destroyed when NewsPlex terminates. |
| database | The database directory contains the article database. It contains one file for each news-group offered by NewsPlex. Each file has one line per article. It describes the article and indicates which servers have it. Do not modify the files there. |
| explore |
The explore directory stores the log files resulting from the
exploration of servers. Log files are named after servers.
Each server exploration has the same scenario, which is logged into the file:
|
| journal | The journal file logs certain events and conditions as NewsPlex runs, like starts of server updates and incoming connections from client news-readers. This file is renamed to .bak each time NewsPlex is started and this file is over a certain size limit and the previous .bak, if any, is deleted. |
| list | The list directory temporarily stores the full news-group listings from news-servers which cannot return news-groups matching patterns. It is automatically destroyed when NewsPlex exits. |
| logs | The logs directory hosts the various output files that NewsPlex produces. |
| logs/cliXXX | The logs/cliXXX files log commands issued by your client news-reader and the responses it got. XXX is a unique number. Once a client news-reader has disconnected, the contents of its logs/cliXXX file are appended to logs/clients and the logs/cliXXX file is deleted. |
| logs/clients | The logs/clients file logs closed sessions from client news-readers. It can be deleted at any time. |
| logs/groups.new | The logs/groups.new file logs the new news-groups appearing in NewsPlex. It is presently written but never read, and can be deleted at any time. |
| logs/results |
The logs/results file indicates the exploration
results for servers. Each line has this format:
server-url [flags] # result (comment)
This file is kept when NewsPlex terminates. It can be deleted
at any time.
|
| logs/retrievd | If the -R command-line option is passed, NewsPlex stores in the logs/retrievd file the Message-ID fields of all the articles which have been successfully retrieved by client news-readers. This file is also used when the -M command-line option is specified (see section below). |
| logs/unusable | The logs/unusable file is only created in Minimal mode (-m command-line option). It contains the logs of server explorations for all servers which did not have any interesting news-groups or which could not be contacted at all. It can be deleted at any time. |
| newsrc |
The newsrc directory contains one file per news-server used.
It logs which news-groups have been explored on the
server and which article descriptions have been retrieved
from them.
Note that the format of the files in the newsrc directory and the information in them are not quite the same as for the newsrc or .newsrc files used by news-readers. You can delete a file corresponding to a server to force NewsPlex to retrieve the full article list for each news-group at the next exploration of the server. This will however almost certainly create dangling references in the article database. Such dangling references can be removed with the NPCHECKGROUPS command, preferably after the server has been re-explored. |
| panic | The panic file is created if NewsPlex detects an non-recoverable internal error. Once a short message has been logged into this file, NewsPlex exits. |
It also adds a few of its own, which are not intended for usage by client news-readers but rather by an operator. These commands can be entered after performing telnet localhost 119 on your computer.
| |
|
| NPACCESSED | Display which news-groups are currently being accessed by client news-readers or are being updated. |
| NPCHECKGROUPS [mode] |
Check the article database for coherency with the newsrc
files and optionally remove dangling article references.
mode specifies the operation mode:
0 - check only, silent mode
If a server is being explored when this command is entered, no checking will be performed for it. |
| NPCLOSE server-number | Close all idle connections to a given server. |
| NPCONNECTED | Display which news-servers are currently being connected to. This can be either for exploring or for retrieving an article on client news-reader request. Connections are maintained a few minutes even when idle, so as to minimize the time necessary to re-use a server. |
| NPDELETEOLD days | Delete descriptions for articles older than days days in all groups. Articles which do not have a recognizable Date will be kept. |
| NPDELETESERVER server-number |
Delete a server which is no more usable.
If a server becomes unusable, as the explore/server-name file indicates, you can delete it from NewsPlex using NPDELETESERVER server-number. NPGLOBALSTATSwill then not show any articles for it anymore. Then you can delete it from the config/servers file. |
| NPEXPLORESERVER server-number | Explore a server immediately rather than waiting till its time comes. This command is also useful if NewsPlex has been started with automatic update disabled (-u). |
| NPGLOBALSTATS |
Display provider stats for all news-groups.
NewsPlex will display for each server how many articles it has and how many of them are unique. If a given server does not appear in the list, NewsPlex has no articles from it. |
| NPGROUPSTATS | Display provider stats for the current news-group. After selecting a news-group with the GROUP name command, you can enter NPGROUPSTATSto see the servers which contain articles for this news-group. |
| NPMARKRETRIEVED article-number | Mark article article-number in the current news-group as retrieved (see section below). |
| NPRENUMBER level |
This command allows to help dealing with unavailable
servers.
If no server having an article is reachable, by
default NewsPlex will send to the client news-reader
500 No server having article is reachableSome client news-readers however do not like this message and do not request further articles. If NPRENUMBER is activated, NewsPlex will instead send 423 no such article number in this news-groupwhich makes the client news-reader think the article has simply been deleted, so further articles will be retrieved correctly. NewsPlex will however create a new reference for the article which will be seen by the client news-readers the next time they asks for new articles. There are 2 possible ways for creating this new reference:
|
| NPRESTART | Shutdown NewsPlex and restart it immediately. All configuration files will be re-read. The original command-line options will remain active. |
| NPSERVERS | Display information about the usage of servers. For each server, NewsPlex will display whether it has already taken article descriptions from it during the current session or was able to connect at all. Other information is the time since the last exploration, the current speed, the number of articles and bytes retrieved. |
| NPSHUT | Shutdown NewsPlex. NewsPlex will terminate. |
| |
|
| -b | Run in background.
By default, NewsPlex runs in foreground and blocks the window from which it has been started. On certain operating systems, the -b option makes NewsPlex put itself into the background. On OS/2, it is not used. Please use the DETACH command of the command processor to start NewsPlex in the backgroud. |
| -d dir | Use directory dir as work directory for files, rather than the directory from which NewsPlex is started. |
| -e | Enable unsecure mode and allow remote connections. By default, only connections from the machine running NewsPlex are possible. |
| -h | NewsPlex will print a short help about command-line options and exit. |
| -i | Shutdown NewsPlex if the Internet connection is lost. This option works only if auto-updating is active (-u not specified). |
| -m |
Perform minimal exploration of servers.
This command causes NewsPlex to run in a special Minimal mode, and only
rate the news-servers listed in the config/servers file.
NewsPlex will try to connect to each of them, and retrieve an article head from each available news-group. If no article could be retrieved, the entire session will be logged into logs/unusable. Otherwise, the session log will stay in explore/server-name. No database or newsrc directories are used or created. |
| -n | Renumber articles if server unreachable. This is the equivalent of the NPRENUMBER command. Specifying this option once causes the renumbering level to be set to 1, and specifying it twice causes the level to be set to 2. |
| -p port | Use port port for establishing the NNTP server. By default, NewsPlex uses the standard port 119. |
| -r |
The list of servers is raw.
If this option is specified, NewsPlex expects a simpler format for the config/servers list of servers, without unique server-numbers in first column: server-url [flags] [# comment]Entries with the same server-url are merged: each server is explored only once even if present multiple times in the list. It is possible to use the logs/results file as a raw config/servers file for the next exploration. |
| -s | Do not establish an NNTP server. |
| -t |
Shorten file names to 8.3 conventions.
This options makes NewsPlex create files following the MS-DOS naming convention, so that NewsPlex can run on filesystems which do not support long filenames. It is not fully implemented yet, since in the database directory filenames still reflect the news-group names. |
| -u | Do not auto-update the article database. It is still possible to explore a specific server by using the NPEXPLORESERVER command. |
| -x |
Do not use the XGTITLE NNTP protocol reequest
for getting the news-groups list.
NewsPlex normally does not use XGTITLE for getting the list of news-groups matching a pattern. However, if the server does not support retrieving names of news-groups using wildcards, then XGTITLE will be used. Since this command is often broken and does not return all the possible news-groups, the -x option allows to avoid using it. NewsPlex will instead get the full news-groups list and filter locally. |
| -D | If this option is specified, NewsPlex will look at the dates of the newsrc files and defer the exploration of servers for which the date of their file is recent. If you restart NewsPlex after a short interruption, it is better to specify this option. |
| -I |
Assume that the Internet is always present. Use this option
if NewsPlex is unable to correctly detect whether you are
connected to the Internet at a given moment, and notably
thinks that you are not while you really are.
Using this option will clutter the logs more easily with uninteresting host unreachable information, and also will cause the renumbering (see the NPRENUMBER command) to be performed when you try to request an article while not being connected to the Internet. |
| -M | This option makes NewsPlex read the logs/retrievd file on startup and mark as retrieved (see section below) all the articles whose Message-IDs are listed in this file. |
| -R | Enable the marking of retrieved articles in the article database (see section below). This option also enables the logging of the Message-IDs of retrieved articles into the logs/retrievd file. |
| -S | Skip groups which appear as empty in the global group list when exploring news-servers. By default, NewsPlex tries to explore groups which appear as empty by other means, in order to find all the articles they might contain in spite of possible incoherencies in the information provided by the news-server, but this is not worth it most of the time. |
| -T count | Explore count servers at once instead of default. By default, up to 8 servers are explored at once in normal mode, and 16 in minimal mode. This is suitable for PPP modem connections. Increasing this number can speed up things up if you have a fast internet connection. Otherwise, it could make things worse because of internet connection contention and resulting timeouts. Also, the more servers are explored at once, the more CPU and memory are used. |
To delete a server, for example because it has been unusable for a certain time, first delete the article descriptions which still point to it, using the NPDELETESERVER command, then delete or uncomment its entry in the config/servers file and restart NewsPlex.
If you start NewsPlex with the -R option, NewsPlex will change the description of an article in its article database after this article has been successfully retrieved for the first time by a client news-reader. It will add a "/r " prefix to the From field of the article description. If an article was previously from John Doe <john@doe.com> it will become an article from /r John Doe <john@doe.com>.
The articles themselves are not modified when they are actually retrieved.
If a client news-reader later requests a full list of articles for a news-group, it will get the modified descriptions. Normally, news-readers only request new article descriptions, so they do not see the modified descriptions unless they request everything.
NewsPlex has a second way of remembering which articles have already been retrieved: it stores their Message-IDs into the logs/retrievd file. This is also enabled with the -R command-line option. It allows to keep the information even after the article has vanished from all servers where it was present, and hence the article description, containing the "/r " prefix has been deleted.
The -M command-line option makes NewsPlex read the logs/retrievd file on startup or restart and mark with "/r " all the articles whose Message-IDs are listed in this file.
This option is useful if you add a new server to the config/servers file and it has articles which you have already retrieved in the past, but which do not exist on any other server anymore. Except for the date, such articles would appear as new, because NewsPlex does not keep information about articles once they disappear from all servers. If the -M option is specified, NewsPlex will mark these old articles with "/r " and you will be able to avoid retrieving them again.
There is presently no way of doing the opposite, that is transferring the Message-IDs of previously retrieved articles from the article database into a file.
typedef enum {
ERROR_OK, /* No problem */
ERROR_HELP, /* Help was printed */
ERROR_BAD_OPTION, /* Bad option was passed */
ERROR_NO_WORK_DIR, /* Missing work directory */
ERROR_NO_JOURNAL, /* Could not open the journal file */
ERROR_ANOTHER_SERVER, /* Another server already running on port */
ERROR_NO_CONFIG, /* The config directory does not exist */
ERROR_NO_GROUPS_YES, /* Missing config/groups.yes file */
ERROR_NO_LOGS, /* Could not create the logs directory */
ERROR_NO_LIST, /* Could not create the list directory */
ERROR_NO_EXPLORE, /* Could not create the explore directory */
ERROR_NO_LONGNAMES, /* -t not specified but disk is 8.3 only */
ERROR_NO_NEWSRC, /* Could not create the newsrc directory */
ERROR_NO_DATABASE, /* Could not create the database directory */
ERROR_NO_UNUSABLE, /* Could not create the logs/unusable file */
ERROR_SERVERS, /* Error in the config/servers file */
ERROR_GROUPS_FILES, /* Error when processing the groups files */
ERROR_NNTPSERVER, /* Could not establish an NNTP server */
ERROR_MAX /* Dummy end of error list */
} eStartupErrors;