ward/ripnews
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

new Mark Remaining Read option

Browse source
This commit is contained in:
Ward Wouts 2005-05-10 20:52:58 +00:00
parent e827c94393
commit 6c92fcd40b
3 changed files with 34 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

6
trunk/ripnews/CHANGELOG
View file

@ -1,6 +1,10 @@
# $Dwarf: CHANGELOG,v 1.37 2005/02/05 08:29:36 ward Exp $
# $Dwarf: CHANGELOG,v 1.38 2005/02/09 09:14:35 ward Exp $
# $Source$
from 0.5.2 to ...
- fix the occasional deadlock
- implement Mark Remaining Read option
from 0.5.1 to 0.5.2
- major improvement in memory usage
- speed ups

12
trunk/ripnews/README
View file

@ -1,4 +1,4 @@
# $Dwarf: README,v 1.19 2005/02/24 09:02:24 ward Exp $
# $Dwarf: README,v 1.20 2005/03/01 09:18:25 ward Exp $
# $Source$
Ripnews is a bulk downloader for usenet. It's quite flexible in terms of
@ -118,13 +118,21 @@ Supported config options:
OPT_I=<pattern> Set include pattern.
OPT_L=<bool> Set long filenames.
OPT_C=<bool> Sets combined filenames.
OPT_X=<pattern> Set exclude pattern. Ripnews will read articles matching this pattern but it will not attempt
OPT_X=<pattern> Set exclude pattern. Ripnews will read articles
matching this pattern but it will not attempt
to download them.
OPT_MR=<pattern> Set "mark read" pattern. Ripnews will place
articles matching this pattern in your newsrc,
afterwards they will never be present in memory
again. Great for reducing memory usage when
checking a group for the first time.
OPT_MRR=<bool> Mark Remaining Read. If this is set to
true and the article doesn't match an exclude or
include pattern, the article will be
marked as read. The purpose of this is
to keep the caches of extremely large
groups small as to make processing
quicker.
OPT_T=<bool> Set test mode. Newsrc files will not be written
to.
TEMPDIR=<dir> Set tempdir location.

25
trunk/ripnews/ripnews.rb
View file

@ -1,6 +1,6 @@
#!/usr/local/bin/ruby -w
# $Dwarf: ripnews.rb,v 1.102 2005/03/01 09:18:50 ward Exp $
# $Dwarf: ripnews.rb,v 1.103 2005/03/01 19:56:56 ward Exp $
# $Source$
#
@ -428,9 +428,9 @@ def get_multi(subj, group)
end
@decode_threads << Thread.new(body, file, fileout, subj) do |tbody, tfile, tfileout, tsubj|
puts "inside thread pre pass\n"
# puts "inside thread pre pass\n"
Thread.pass
puts "inside thread post pass\n"
# puts "inside thread post pass\n"
if UUEncode.is_uuencoded(tbody)
print " UUDecoding...\n"
if tfile
@ -478,7 +478,7 @@ def get_multi(subj, group)
output_data(tsubj, tmode, tfilename, tbody)
end # thread end
puts "ouside thread\n"
# puts "ouside thread\n"
return true
else
@ -488,7 +488,7 @@ def get_multi(subj, group)
end
def fill_preselector(group)
if @config[group].has_key?("-I")
if @config[group].has_key?("-I") and !(@config[group].has_key?("-MRR") and @config[group]["-MRR"])
@articles.set_preselect_pattern(Regexp.new(@config[group]["-I"]))
end
end
@ -652,12 +652,12 @@ def main
end
for i in @articles.get_group_subjects.sort{|a, b| ward_sort(a, b)}
print "#{i}\n" if Debuglevel > 2
# explicitly mark as read
if @config[group].has_key?("-MR") and i =~ /#{@config[group]["-MR"]}/
print "Marking '#{i}' as read\n"
@articles.group_update_newsrc(i)
next
end
if !(@config[group].has_key?("-X") and i =~ /#{@config[group]["-X"]}/) and
# get the juicy bits
elsif !(@config[group].has_key?("-X") and i =~ /#{@config[group]["-X"]}/) and
i =~ /#{@config[group]["-I"]}/
print "Match: #{i}\n" if Debuglevel > 0
if @articles.group_is_complete(i)
@ -667,21 +667,24 @@ def main
elsif @articles.group_is_multipart(i)
get_multi(i, group)
end
#rescue Article::TempError, Article::PermError
rescue TempError, PermError
print "#{$!}\n"
print " Skipping article...\n"
#print "Caught #{$!.class}\n"
#print "Error: #{$!}\n"
next
end
else
print "Not complete: #{i}\n"
end
# if Mark Remaining Read is set do so
elsif @config[group].has_key?("-MRR") and @config[group]["-MRR"] and
!(@config[group].has_key?("-X") and i =~ /#{@config[group]["-X"]}/)
print "Marking remaining '#{i}' as read\n"
@articles.group_update_newsrc(i)
end
end
# hier wachten op evt. threads...
# wait for threads if there are any
if ! @decode_threads.empty?
@articles.disconnect
puts "Waiting for decode threads..."