java - Crawling data from Web, When page loads data dynamically at end of page -

-

I want to crawl data from some web to Java, but I have found that the page is loading when the page's Reaches the end. I'm not a web developer and do not know what technology they use to load data while reaching the end of the scroll page.

Can you give me some hints? What technology did they use? How can I read the data when I do not want to use the browser? (I wrote a code with Java using urlConnection to read data from the site.

The site is something like this.

Thank you. < P>

This is a common "problem" of web crawler bots ... some pages have dynamic content added from the sources contained in it. Loads or triggers (such as your example - by scrolling down) can be loaded. When the target page downloads and scraps the dome structure In most cases, the HTML elements of the external data are not included.

What I suggest to do is to identify the source path of this data, which is carefully coded on the dome And it is being called as a secondary source, which includes all the necessary data.

Edit:

For example you linked- Its easy: 

  - Install Firebug - S Scroll down the page to check the Crypt - request puts the fire - now you can see the links and wars used to add dynamic as content.   
 www.healthtap.com/#topics/Women%27s%20health:  
 Dynamical Scale Feedback Link:  
? Extended categories = 1 & amp; Auth_token = false & amp; Per_page = 8 & amp; Page = 7 & amp; Per_page = 8 & amp; Auth_token = false & amp; Gener_token = true  
 As you can see here some parameters you can play on:  
  1 / topic / + Page Fir Name of Name + Jason? 2 / per page = num - & gt; How much results 3 / gener_token = true - & gt; To return it is a security value, but it changed to false and it works fine ....   
 Now you can play with this link and load all the data You can merge it from the main page and you can crawl it.  
 Test!

Comments

Post a Comment

Popular posts from this blog

excel vba - How to delete Solver(SOLVER.XLAM) code -

-
After opening several examples of macro / code written, I suddenly get stuck with a code that seems safe. Solver (SOLVER.XLAM) is doing this notation and I would like to remove it from my system. I do not think this is a relativant code. Can anyone tell me what should I do? I have already installed something to remove the password but Solver is still asking for password and I can not get rid of it! itemprop = "text"> Solver Excel is add-in If you want to delete it, then Excel Application & gt; Options & gt; Add-ins & gt; Search for 'Go to' button and gt; Uncheck the solver on the list like press something (I can not give you the exact path because I have the Polish version of Excel). Trying to open Solver's VBA code is not good because it There is allegedly copyright. Besides, I do not think Solver creates any disadvantages while working with Excel or VBA.
Read more

github - Teamcity & Git - PR merge builds - anyway to get HEAD commit hash? -

-
I have a team city build project with Gitbab VCS route, I have been triggered for both / head on PR and / Merge the referee. The annoying thing is that you can not do anything useful with the merged hash - it is not present in Gitobb. I want to exclude newsgate packages with head in gutub comet situations in the version number along with Hewash (Newsaging 7) for the music build, but against head hash. However, I can not see a way to get the head hashes while constructing that merge. I just want to head because it's useless - we only care about merge. (PR - Pull Request) This command works reliably while running One on / merge is git log --no-merge -1 ---- = '% h'
Read more

ios - Replace text in UITextView run slowly -

-
I have a UITextView and I change a word like the code at the end of UITextView.text: - (void) Replace TexttechWitWord: (NSString *) word {NSString * currentTextViewText = self.textView.text; CurrentTextViewText = [currentTextViewText stringByReplacingCharactersInRange: NSMakeRange ([self.textView.text length] - [word length], [word length]) withString: word; Self.textView.text = currentTextViewText; } Replacement work is fine, but it runs very slowly and very slowly when the textView now lengthens Is there any better way? I try the replacement range: with text for a week but still stuck. Not to know how to use it to replace it in your case. Please guide! Thanks in advance! This is an important job. As you are using an irreversible string and you are creating a very long string and before Assigned to NSString created from. NSString will be used instead of NSMutableString . - (UITableViewCell *) TableView: (UITableView *) Table View CellForOutPath: (NSIndex...
Read more