python - How to use the Rule class in scrapy -

-

i trying use rule class go next page in crawler. here code

from scrapy.contrib.spiders import crawlspider,rule scrapy.contrib.linkextractors.sgml import sgmllinkextractor crawler.items import gdreview   class gdspider(crawlspider):     name = "gd"     allowed_domains = ["glassdoor.com"]     start_urls = [         "http://www.glassdoor.com/reviews/johnson-and-johnson-reviews-e364_p1.htm"     ]      rules = (          # extract next links , parse them spider's method parse_item         rule(sgmllinkextractor(restrict_xpaths=('//li[@class="next"]/a/@href',)), follow= true)     )       def parse(self, response):         company_name = response.xpath('//*[@id="eihdrmodule"]/div[3]/div[2]/p/text()').extract()          '''loop on every review in page'''         sel in response.xpath('//*[@id="employerreviews"]/ol/li'):             review = item()             review['company_name'] = company_name             review['id'] = str(sel.xpath('@id').extract()[0]).split('_')[1] #sel.xpath('@id/text()').extract()             review['body'] = sel.xpath('div/div[3]/div/div[2]/p/text()').extract()             review['date'] = sel.xpath('div/div[1]/div/time/text()').extract()             review['summary'] = sel.xpath('div/div[2]/div/div[2]/h2/tt/a/span/text()').extract()              yield review

my question rules section. in rule, link extracted doesn't contain domain name. example, return "/reviews/johnson-and-johnson-reviews-e364_p1.htm"

how can make sure crawler append domain returned link?

thanks

you can sure since default behavior of link extractors in scrapy (source code).

also, restrict_xpaths argument should not point @href attribute, instead should either point a elements or containers having a elements descendants. plus, restrict_xpaths can defined string.

in other words, replace:

restrict_xpaths=('//li[@class="next"]/a/@href',)

with:

restrict_xpaths='//li[@class="next"]/a'

besides, need switch lxmllinkextractor sgmllinkextractor:

sgmlparser based link extractors unmantained , usage discouraged. recommended migrate lxmllinkextractor if still using sgmllinkextractor.

personally, use linkexractor shortcut lxmllinkextractor:

from scrapy.contrib.linkextractors import linkextractor

to summarize, have in rules:

rules = [     rule(linkextractor(restrict_xpaths='//li[@class="next"]/a'), follow=true) ]

Comments

Post a Comment

Popular posts from this blog

c++ - Delete matches in OpenCV (Keypoints and descriptors) -

-
i want check scene image against 2 train images. that, detect features , compute descriptors of both training images. before detecting, computing , matching scene image, delete matches of train1 , train2. because these matches won't facilitate matching of scene image train1 , train2. so, match train1 train2 , vector of matches trainidx , queryidx. how can delete these matches in keypoints-vector , descriptor matrix of train1 , train2? best regards, dwi i have done below: std::vector<cv::keypoint> keypoints[2]; cv::mat descriptor[2]; std::vector< cv::dmatch > matches; /* write code generate keypoints, descriptors , matches here... keypoint[0] -> train image 1 keypoints keypoint[1] -> train image 2 keypoints descriptor[0] -> train image 1 descriptors descriptor[1] -> train image 2 descriptors matches -> matched between train image 1 , 2 */ // logic keep unmatched keypoints , corresponding descriptor
Read more

java - Could not locate OpenAL library -

-
i writing game using slick2d library, uses lwjgl. lwjgl version 2.9.3. using netbeans 8. i tried adding sounds game, example music track on main menu. problem is, exception when run game, despite game still compiling , running except sound of course: sat mar 21 01:23:57 edt 2015 error:could not locate openal library. org.lwjgl.lwjglexception: not locate openal library. @ org.lwjgl.openal.al.create(al.java:156) @ org.lwjgl.openal.al.create(al.java:156) @ org.lwjgl.openal.al.create(al.java:102) @ org.lwjgl.openal.al.create(al.java:206) @ org.newdawn.slick.openal.soundstore$1.run(soundstore.java:295) @ java.security.accesscontroller.doprivileged(native method) @ org.newdawn.slick.openal.soundstore.init(soundstore.java:292) @ org.newdawn.slick.music.<init>(music.java:156) @ org.newdawn.slick.music.<init>(music.java:75) @ view.mainmenu.init(mainmenu.java:58) i didn't include rest of error states error happens in game code. the
Read more

sorting - opencl Bitonic sort with 64 bits keys -

-
i used nvidia sdk bitonic sort , works great me. 32 bits (uint) need ulong keys. have typically 2^14 keys @ time , power of 2. searched on not find kernel designed ulong. i tried modified nvidia sdk bitonic sort using ulong keys not work. kernel not crash after de call clenqueuendrangekernel got error : cl_invalid_command_queue can tell me how modify bitonic sort or radixsort, or can sort ulong keys? i running nvidia quadro 4000 opencl 1.1 cuda 6.5.20 full_profile used original nvidia sdk bitonicsort.cl i used"ulong" keys , value input , ouput instead of "uint" thanks helping i made work-around suit need. used 30 bits of key , 18 high bits of value sorting. changed comparators (2 lines): '#define mask 0xffffc000 // suit need from this: if( (*keya > *keyb) == dir ) this: if(((*keya > *keyb)||((*keya == *keyb)&&((*vala & mask)> (*valb & mask)))) == dir ) it works fine. but, still, it's work-a
Read more