Found a solution.
The problem was, that when scrolled down the feed, new elements were displayed and some of them had another RecyclerView inside and since I was searching by class name, both recycleViewers were found.
What I did was to search by resource-id and not by class name:
List tempElements = driver.getDriver().findElements(By.xpath(".//*[@resource-id=’" + idOfRecyclerView + "’]/child:: * "));