I've lost my phone twice, so I know exactly what it feels like to be phoneless (for a short period of time).
I think the hardest thing to deal with is the anxiety of not being connected. I routinely check emails, social media accounts, news feeds, Words With Friends throughout the day, so breaking that habit was tough...
I even reach for my phone and do those things without even realising it sometimes!
The other thing to worry about is whether people are trying to get in touch with you. What if something urgent happens at work or at home, and you need to be informed. Without a phone, being alerted to problems and emergency situations becomes difficult.
Finally, there's all the data we store on our phones to consider... Photos, documents, videos, our entire lives really!
Yes, there are cloud services and backup options in some cases, but what about the risk of that information falling into the wrong hands? That's something that worried me when I lost my phone, and changing passwords and making everything secure after a lost phone can also be time consuming.
Then again, I hear a lot of youngsters complaining about how their parents won't buy them a phone, and I think back to my younger days... We didn't have cell phones until I was in my late teens, so it's certainly possible to survive - but it meant more payphone visits (and payphones aren't as abundant these days), and there is more to miss out on nowadays I guess.