In 1929 when his father informed him that he would have to leave home. Times were tough and there was only so much food to go around. "Go fend for yourself," Lee's father instructed him. "I cannot afford to have you around any longer." Lee reluctantly hopped a freight car out of Louisiana in hopes of realizing better days. What he found instead was the prevailing loneliness of vagabond life: wandering from place to place looking for work, food, and shelter. Added to his plight was the fact that Lee's skin color made him suspect in the eyes of many of the folks he met along the road. He recalled how white vagabonds were treated differently than African American vagabonds when it came to soliciting the kindness of strangers. "White kids, they fared better. They might let them stay in a house with them, but me, I could sleep in a barn with the mules and the hay."

While tales of friendships among vagabonds that transcended race abound, many African American vagabonds recounted being made to feel like outcasts among outcasts. Harold Jeffries of Chicago recalls how he first felt the sting of racism when he and six buddies took to the tracks out of Minneapolis in 1935:

"As black kids from the north, we'd heard of racial discrimination, but not one of us had actual experience of harsh prejudice. Our first frightening encounter came at the Union Pacific roundhouse in Kansas City. Some of the kids drank from a 'whites only' fountain. We were literally run out of the U.P. yards. We walked across the long bridge from the Kansas side of the city over to Missouri to see a girl I'd met back home. We told her family what had happened in the U.P. yards. They sat us down and gave us a lecture about the ways of the South."

Sadly, one of the "ways of the South" was the brutal practice of lynching. As the Great Depression rocked the economies of every state in the Union, jobs were scarce. Among African Americans, unemployment reached a staggering fifty percent. Even so-called menial jobs were now eagerly sought after. In many cities, whites demanded that they be given first crack at any job openings by virtue of their skin color. Some went so far as to revive lynchings as a way of intimidating employers and African Americans looking for work. The New Republic, reporting on the increase in the number of lynchings in the South, observed that "dead men not only tell no tales but create vacancies." Lynchings were also used to enforce mob justice. An African American vagabond had to be aware of the racial climate in the towns he was traveling through. Sometimes, they were tipped off. Clarence Lee recalls:

"I was leavin' from Baton Rouge to go to Denham Springs, Louisiana, and this man made one stop in between. He had a small station place and somebody got on the train and was talking to the conductor and he says, 'Well, that boy has to be put off here. They're going to lynch him. See, there has been a rape between here and Denham Springs, and he fits the description.' So he put me off right in the middle of a swamp. Probably saved my life."

Through his fiction, esteemed author Ralph Ellison recounted his experiences as an African American riding the rails. In 1933, Ellison earned a scholarship to the Tuskegee Institute to study music. Because he could not afford train fare to make the journey from his home in Oklahoma City to Alabama, he resorted to hopping freights. In his short story, "I Did Not Learn Their Names," he writes of confronting fellow vagabonds whose bigoted attitudes where passed down to them from their elders. "I was having a hard time trying not to hate in those days, and I felt bad whenever I found myself in a position that might have been interpreted that way. But I had learned not to attack those who were not personally aggressive and who only expressed passively what they had been taught." Still, Ellison's acquiescence would only go so far. As a fellow traveler among the wandering and the wounded, he refused to quietly take his "place." He wrote, "I had learned that on the road you really had no 'place'; you were all the same though some of them did not understand that."

For many vagabonds, however, life on the road — and often on the run — engendered a greater understanding of the plight of African Americans and others looked down upon by certain sectors of society. "Steam Train" Maury Graham, who began his vagabonding days in 1931 when he was fourteen, points out that "having experienced the intolerance of those who despised hobos, the 'bo tended to make the acceptance of others — no matter what their color — a legacy that they hoped would be passed on to everyone."